Sign in to compare your saved sessions.
{{ state.message }}
{{ state.benchmark ? 'No rated one-shot results yet.' : 'Compare two random sessions from the saved sample. Judge how useful the work was for its request.' }}
{{ state.benchmark ? 'Which first scene result is better?' : 'Which agent did the better job for what was asked?' }}
{{ identity(state.pair[side]) }}
{{ state.mediaErrors[side] }}
Quality {{ state.pair.ratings[side].quality }}/5 · {{ state.pair.ratings[side].complaints.length }} complaints · {{ state.pair.ratings[side].acceptable ? 'Accepted' : 'Needs fixes' }}
{{ minutes(state.pair[side].duration_seconds) }} to first result · Production process
Saved: {{ choiceLabel(state.pair.choice) }} · Identities revealed.
{{ run.case_id }} · {{ run.state }} · {{ run.error }}
| Configuration | Rated | Accepted | Complaints | Quality | First result | Elo |
|---|---|---|---|---|---|---|
| {{ identity({configuration:row.configuration}) }} | {{ row.rated }}/{{ row.attempts }} | {{ row.one_shot_acceptance == null ? '—' : Math.round(row.one_shot_acceptance*100)+'%' }} | {{ number(row.mean_complaints) }} | {{ number(row.mean_quality) }} | {{ minutes(row.median_seconds) }} | {{ row.elo == null ? 'Unrated' : row.elo }}provisional · {{ row.matches }} comparisons |
Watch and judge the finished scene as a whole: story, sprites and sets, voices, movement, music, effects and final mix. One fresh production run receives the same source and brief and creates its own script, artwork, narration and sound. Internal self-review is allowed; your feedback comes after the first finished film is frozen. Complaints may concern any part of the scene. Compare only attempts of the same case.
Human quality, acceptance, complaints and preferences determine the result. The production session becomes visible after voting, so you can investigate the models, decisions and waits behind a flaw. First-result time includes the complete production and rendering. Failed or timed-out attempts count against acceptance; completed but unrated attempts remain unknown. Elo starts at 1000 with K=32, counts ties and excludes “Can’t compare”. Ratings under ten comparisons are provisional. The old fixed-art/narration staging prototype is excluded.
Saved request and answer excerpts, not complete transcripts or verified implementations. Model, effort, links and automated scores stay hidden until you vote; recognizable wording can still give clues. Different tasks may be incomparable. Later-session flaws: unknown in v0. No automated follow-up search or model ranking from these choices.
What we’ll do with your choices · Turn timing, cache and session diagnostics ↗
These are model judgments, separate from your choices. Opening them reveals model identities and may influence later comparisons.
{{ state.status }}
Rejudge {{ data.run.status }} · {{ data.run.completed }}/{{ data.run.total }} operations completed · {{ data.run.step }} · Judge: {{ data.run.judges.join(', ') }} · {{ data.run.error }} · Ratings may contain verdicts from the previous run.
| Model · effort | Rating | Requests | Mean difficulty | Mean success | Median API estimate | Median time | $ per success | Quality first | Balanced |
|---|---|---|---|---|---|---|---|---|---|
| {{ a.model }} · {{ a.effort }} | {{ a.rating }}provisional | {{ a.matches }} | {{ a.mean_difficulty }}/10 | {{ a.mean_success }} | {{ a.median_cost_usd == null ? 'Unavailable' : '$' + a.median_cost_usd.toFixed(2) }}{{ a.priced_requests }} priced | {{ a.median_minutes }} min | {{ a.usd_per_success == null ? '—' : '$' + a.usd_per_success.toFixed(2) }} | {{ a.tradeoff ? a.tradeoff.quality_first.toFixed(2) : '—' }} | {{ a.tradeoff ? a.tradeoff.balanced.toFixed(2) : '—' }} |
Elo from judged requests in your own sessions, each played against its difficulty rating. Cost is an API list-price estimate, not subscription spending or measured quota. Missing usage, tier, or required rates makes a request unpriced. Composite scores require every rated request to be priced. Account quota changes may include concurrent sessions; they are not attributed session costs. Quality first and Balanced weight rating, time and cost 1 / 0.2 / 0.2 and 1 / 1 / 1.
This diagnostic never changes ratings, costs, time or rankings. It compares quota used before and after a recorded session, separately for each window. Overlapping sessions are N/A. Because account slots can share a subscription, any recorded overlap on the same provider is conservatively excluded, including idle gaps and Claude activity from connected hosts.
Both readings must be within two minutes of the boundaries, share a reset, and show no intervening reset or decrease. Unrecorded machines, account switches within one slot, delayed updates and rounded meters remain limitations. These are observed percentage-point changes, not exact charges or equivalent allowances.
Quota check has not been collected yet.
{{ data.quota_sanity.error }} · Showing the previous check.
Source snapshot: {{ data.quota_sanity.source_at }} · Checked: {{ data.quota_sanity.checked_at }}
{{ q.status === 'observed' ? 'Observed' : 'N/A' }} · {{ q.provider }} · {{ q.session }} · Slot: {{ q.accounts.join(', ') }}
{{ quotaReason(q.reason) }} ({{ q.overlap_count }} other sessions)
{{ q.start_at }} → {{ q.end_at }}
{{ w.window_minutes }}-minute window: {{ w.before_used_percent }}% → {{ w.after_used_percent }}% used · {{ w.delta_pp }} percentage points. Readings: {{ w.before_at }} → {{ w.after_at }}. Reset: {{ w.resets_at }}.