How every result on this site was produced and labelled. ingest.py renders this file into the methodology page and the home-page summary, and publishes it in data/content.json (one entry per ## section).
Home harness: each vendor's own CLI (Claude Code for Anthropic, Codex CLI for OpenAI, dsh for DeepSeek, Grok Build for xAI, Gemini CLI for Google, Kimi Code CLI for Moonshot). GLM-5.3 has no coding CLI of its own, so its home track runs in Claude Code, Z.ai's recommended harness, and is labelled that way. Neutral: every model in OpenCode with the same configuration. Raw baseline: an earlier single API request with no harness and no tools. Baselines are kept for reference only; the project's rule is now harness-only. Provider = the API service that served the model for that run. Gemini 3.8 Flash stays in the roster and in the planned totals, but none of its cells is tested: we have no usable access to the model.
" }, { "id": "max-effort", "title": "Max effort", "html": "Every model runs at the highest reasoning effort its API accepts (max for Claude and DeepSeek, xhigh for GPT-6 Astra and Grok 4.7, high for Gemini 3.8 Flash). A few raw baselines ran at lower effort; the effort is printed on every result.
" }, { "id": "one-prompt-fully-autonomous", "title": "One prompt, fully autonomous", "html": "One prompt, then no human input until the model stops. Inside a harness the model may search the web, render and inspect its own output and fix it (self-review allowed). The single exception, where the reviewer sent written feedback once, is labelled 1 fix turn.
" }, { "id": "whose-failure-was-it", "title": "Whose failure was it?", "html": "DNF means the model failed on its own, for example by spending its whole output budget thinking. When we end a run (our client or safety timeout, or a rule change), it is marked Stopped and not counted against the model. Harness-config failure (ours) means our own harness settings killed the run. Harness + model mismatch means the run is excluded as not a clean single-model result: either the harness and the model could not work together (for example Claude Code sending an image to GLM-5.3's text-only endpoint), or the harness ran part of the job on a different model (a mixed-model run, for example Claude Code spawning a subagent on another model); each run's note says which. Infrastructure DNF means the connection upstream of the harness dropped the run. None of the last three is counted as a model failure; the field notes describe each incident. The spend on every failed run is still counted.
\nAttempts lost to the harness launcher or the connection before a result existed are not results: they are listed under the results as infrastructure failures (not model results) and left out of every count, total per model and head-to-head. A planned cell that is queued but has not run yet says Pending — queued, not run yet.
" }, { "id": "costs-reported-vs-estimated", "title": "Costs: reported vs estimated", "html": "Reported: the harness printed its own bill (for example Claude Code's total_cost_usd or OpenCode's session cost). When a harness reported one, it is the headline cost, labelled as that harness's figure, with the list-price estimate beside it. Estimated: token counts times the vendor's list price, with the price table and its date on every result. Unknown: the tokens were never reported, so we don't guess. When no token usage was recorded at all (for example a stream cut before its usage record), the cost reads no usage recorded, never $0. Money spent on runs that produced nothing is shown, never dropped. Totals say what they include: the result totals cover the results on this site, failed runs included; setup smoke tests and infrastructure failures are totalled separately.
15 OpenCode results ran on a box whose per-request proxy logs were lost; the served model for those runs could not be independently verified. Their per-step costs match the requested model's prices.
" }, { "id": "how-results-are-scored", "title": "How results are scored", "html": "Scores have three separate sources: the AI judge, visitor votes (coming later, with a blind reveal), and the reviewer's grade. The current AI judge is the test runner, model claude-opus-5-5, described as reviewing the 1600 px render against the published, pre-registered rubric. Each scored run currently has one recorded AI-judge assessment. The reviewer has not given any numeric grades yet. Reviewer grades and visitor votes, when available, appear alongside the AI score and are not blended into it.
\nOnly finished runs covered by a published, pre-registered rubric receive a score. Scores run from 0 to 10: up to 8 points for the prompt's requirements and up to 2 for craft, with half points for partial credit. Cost and time do not contribute to this score. Each AI score links to its rubric and remains labelled Provisional — AI judge until the reviewer checks it. Scores originate with the judge or reviewer; the site's build agent does not assign them.
\nWe publish the per-criterion marks and calculate the total from them. When those marks cannot be read into the published breakdown, we retain the judge's stored total and label it Provisional — stored total; criterion breakdown unavailable. The displayed total has not been verified through that breakdown. A prompt's board stays unranked until every published AI score on it has a complete, validated criterion breakdown. Missing breakdowns and any disagreement between marks and totals must be resolved before ranking that board; affected runs remain visible throughout.
\nEligible AI-judge boards are ranked by score within one prompt, highest first. Each row represents a particular run, identified by its model, harness, provider, effort and attempt. Equal totals share a rank, and the next rank skips the occupied positions: 1, 1, 3. Row order inside a tie has no quality meaning. These are provisional ranks of the recorded assessments. They do not establish that a model is generally better, or that a small score difference would persist in another run or under another judge. We do not publish confidence intervals or treat unequal scores as statistical ties without evidence of judging and run variability.
\nScores from different prompts are never averaged or used to produce an overall model ranking. Home and neutral harness conditions stay identified, and raw baselines remain a separate reference section. Each board states how many recorded results are scored, how many are unfinished and how many remain unscored. The reviewer's written recommendations and “best” picks remain separately attributed editorial judgments.
\nDNF is a result. DNF, stopped, failed and no-output results remain visible on the same prompt board, with their recorded status, reason, time and cost. They receive no quality rank and no invented zero score. No output artifact describes the absence of a published deliverable; it does not by itself identify whose failure it was. A stopped run with partial output keeps that output, labelled Work in progress. Finished runs without a score say Not scored, with the reason. Prompts with no scores still have an unranked result board.
\nModel DNF remains distinct from our stops, harness-config failures, harness/model mismatches, infrastructure failures and unexplained errors. All records published as results stay in result totals, including failures. Separately logged infrastructure attempts that are not result records remain visible as Infrastructure failures (not model results) and outside those totals. Pending and no-access cells remain visible as Pending — queued, not run yet or Not tested (no access); neither is a scored run. Unknown cost is never displayed as zero.
" } ], "source": "content/methodology.md" }, "statuses": [ { "id": "ok", "label": "OK", "help": "Finished on its own and produced the artifact." }, { "id": "dnf", "label": "DNF", "help": "Did not finish: the model gave up or used up its own budget (e.g. hit max_tokens while still thinking). Counted against the model." }, { "id": "stopped_by_our_timeout", "label": "Stopped (our timeout)", "help": "Our client or safety timeout ended the run. Not a model failure." }, { "id": "stopped_by_us", "label": "Stopped by us", "help": "We ended the run ourselves (for example after a rule change). Not a model failure." }, { "id": "harness_config_failure", "label": "Harness-config failure (ours)", "help": "Our harness configuration killed the run. Not a model failure and not our safety timeout." }, { "id": "harness_mismatch", "label": "Harness + model mismatch", "help": "Excluded as not a clean single-model result in this harness. Either the harness and the model could not work together (for example the harness sent an image to a text-only model endpoint), or the harness ran part of the job on a different model (a mixed-model run, for example Claude Code spawning a subagent on another model). The run's own note says which. Not counted as a result for the model and not an infrastructure failure." }, { "id": "infra_dnf", "label": "Infrastructure DNF", "help": "Did not finish because the connection upstream of the harness dropped the run (for example a silent stream cut after about 5 minutes). Not a model failure and not our harness configuration." }, { "id": "error", "label": "Error", "help": "Something else broke (a crash or an unexplained failure)." } ], "scoring": { "rule": "Scores come from three sources, shown side by side. The AI judge (the test runner, labelled on every score) scores finished runs against a published, pre-registered rubric; its per-criterion marks are shown and the total is computed from them, labelled a draft score until the reviewer checks it. Visitor votes are coming later. The reviewer's own grade appears when it exists. The site's build agent never scores. AI-judge boards are ranked by score within one prompt, equal totals sharing a rank; scores from different prompts are never averaged.", "not_scored_label": "Unscored", "scored_results": 63, "checklists": [ { "prompt_id": "ps5_controller", "title": "PS5 Pro DualSense controller (max-fidelity SVG)", "checklist": [ "Accurate DualSense silhouette and proportions: wide body, wide touchpad top centre, flared rounded grips, symmetrical, white shell with black core", "Face buttons in the classic diamond with correctly coloured outline PlayStation symbols on light buttons, subtle bevels", "D-pad as FOUR separate arrow-shaped buttons, not a single cross", "Two black analog sticks with concave tops, textured grip ring and shading", "PS logo button centred below the touchpad, mic mute button with orange indicator below it, Create (left) and Options (right) flanking the touchpad", "Blue light-bar glow along both touchpad edges with bloom; speaker grille holes on the black core", "Realistic product-render shading: gradients, specular highlights, ambient occlusion, contact shadow", "Dark studio background with radial vignette; front-on hero shot, centred and filling the frame" ] }, { "prompt_id": "ps5_controller_quick", "title": "PS5 Pro DualSense controller (quick SVG)", "checklist": [ "White main body with a black central section", "Large touchpad on top with blue light-bar edges", "D-pad on the left", "Four face buttons with triangle / circle / cross / square symbols", "Two black analog sticks", "PS button and mic mute button between the sticks; Create and Options buttons", "Curved grips and accurate proportions", "Front-on centred hero shot with gradients, highlights and soft shadows on a subtle dark studio background" ] }, { "prompt_id": "ps5_console", "title": "PS5 Pro console and controller product shot (SVG)", "checklist": [ "Slim PS5 Pro console design", "White side panels with a glossy black central band", "Three horizontal vent slits across the white panels", "Console standing upright", "White-and-black DualSense controller beside it", "Clean studio product shot with a soft gradient background", "Subtle floor reflection and soft shadows", "Accurate proportions, gradients and highlights for realism" ] }, { "prompt_id": "fjord", "title": "Cinematic fjord boat scene (Three.js)", "checklist": [ "One self-contained index.html; Three.js r160 from the pinned import map; all textures procedural", "Procedural fjord walls and mountains with snow caps and height/slope colouring", "Instanced conifer forests, distant islands, a lighthouse with a rotating beam", "Animated water with reflections; a wooden boat with two animated characters", "Weather cycle clear > fog > rain > storm > clearing with smooth transitions, lightning, wind and waves", "Full day/night cycle with sky, sun/moon and fog tracking time; stars, moon, lit lighthouse and lantern", "Keyboard and touch controls, follow camera, cinematic auto-tour, minimal UI", "URL parameters (tour, capture, cycle), deterministic timing, __sceneReady, smooth on phones, no console errors" ] }, { "prompt_id": "js_short", "title": "Animated short film drawn in code (HTML/JS)", "checklist": [ "One file, no external assets; 1920x1080 canvas letterboxed to the window", "Deterministic: requestAnimationFrame + performance.now only, seeded noise, FILM_DURATION, holds the last frame", "Original WebAudio score via buildSoundtrack / renderSoundtrack, with a tap-for-sound hint", "Original story with an emotional arc and one consistent main character in every scene", "At least four distinct scenes with real transitions", "Expressive character animation: walk cycle, blinking, changing emotions", "Virtual camera with parallax depth; lighting and time-of-day changes with grading and glows", "Two or more particle/weather types, storybook subtitles, cohesive art direction, closing title" ] } ] }, "rubrics": [ { "id": "rubric", "title": "Wave-1 SVG scoring rubric (AI AI Model Benchmark)", "covers": [ "pelican_bicycle", "macbook_pro_16", "lighthouse_night", "isometric_town", "pelican_bicycle_animated" ], "received_utc": "2026-09-23 21:25 UTC", "author": "the wave-1 SVG test runner", "url": "methodology.html#rubric", "html": "Written once before any wave-1 result was scored, and kept for every wave-1 SVG card. Score 0–10, comparable only within one prompt. Scores are given only to ok runs; DNF / error / timeout cards carry no score.
Checklist, 8 points — the prompt's own requirements, split into 4 items of 2 points each (1 = partly there, 0 = missing/wrong):
\n| prompt | item 1 (2) | item 2 (2) | item 3 (2) | item 4 (2) |
|---|---|---|---|---|
| pelican_bicycle | reads instantly as a pelican (long beak + pouch, body, wing, legs) | a real bicycle: two wheels, spokes or hubs, a diamond-ish frame, pedals/crank, handlebars, seat | pelican is riding it: sits on the seat, feet on pedals, wing/beak toward the bars; plausible contact points | composition: whole subject in frame, ground/scene context, readable at thumbnail size |
| macbook_pro_16 | recognisable MacBook Pro 16\" (proportions, notch/camera, keyboard + large trackpad, hinge) | slight three-quarter view with a consistent perspective (lid and base share one vanishing logic, no warped keys) | screen glow: lit display with believable content and light spill | aluminium material: gradients, edge highlights, shadows that fake 3D |
| lighthouse_night | lighthouse on a rocky headland, correct structure (tower, lantern room, gallery) | stormy sea with waves breaking; moonlight on the water | beam cutting through rain and mist (volumetric feel, rain streaks) | atmosphere/painterliness: mood, depth, colour harmony |
| isometric_town | sprite sheet first: several distinct building designs in <defs>, reused via <use> | 20+ buildings placed on a true isometric grid, correct z-ordering (back-to-front, no overlaps in the wrong order) | roads, trees, a river and a bridge that crosses it | theme + palette chosen and applied consistently |
| pelican_bicycle_animated | pelican + bicycle drawn well (same bar as the control) | wheels rotate (correct direction) and the crank/pedals turn | legs pedal in sync with the crank (feet stay on the pedals, knees bend plausibly) | seamless loop, SVG/SMIL/CSS only, no JavaScript (a <script> = 0 here) |
Craft, 2 points — polish and finish: clean shapes, no glitches (stray paths, clipped elements, broken filters, text overflow), pleasing colour and detail, renders the same in Chromium as the model intended, file saved under the requested name. 2 = showcase quality, 1 = fine with visible flaws, 0 = rough.
\nscore_note lists the five sub-scores, e.g. pelican 2 · bicycle 1.5 · riding 1 · composition 2 · craft 1.5 = 8.CLAUDE_BYTE_STREAM_IDLE_TIMEOUT_MS=1800000 (30 minutes) and CLAUDE_CODE_DISABLE_NONSTREAMING_FALLBACK=1, including this site's own build sessions.",
"impact": "Claude Fable 5.1 attempt 1 on the DualSense prompt and the first Claude Code fjord attempt with Opus 5.5 are labelled **Harness-config failure (ours)**, not DNF; their spend is counted. This site's build session 01 also lost one turn to it (see [How this site was built](built.html)).",
"impact_html": "Claude Fable 5.1 attempt 1 on the DualSense prompt and the first Claude Code fjord attempt with Opus 5.5 are labelled Harness-config failure (ours), not DNF; their spend is counted. This site's build session 01 also lost one turn to it (see How this site was built).",
"results": [
{
"id": "ps5_controller__claude-code__fable-5-1__attempt1",
"model": "Claude Fable 5.1",
"harness": "Claude Code 2.1.280",
"prompt_id": "ps5_controller",
"status": "harness_config_failure"
},
{
"id": "fjord_claude_code_opus55_attempt1",
"model": "Claude Opus 5.5",
"harness": "Claude Code 2.1.280",
"prompt_id": "fjord",
"status": "harness_config_failure"
}
],
"missing_results": []
},
{
"id": "upstream-silence-drop",
"title": "Streams silent for about 5 minutes are dropped upstream of the harness",
"category": "infrastructure",
"status": "open",
"symptom": "With the watchdog raised to 30 minutes and the fallback off, Claude Fable 5.1's stream still ended after 5 m 16 s of silent thinking: “API Error: Stream ended without receiving any events”.",
"symptom_html": "With the watchdog raised to 30 minutes and the fallback off, Claude Fable 5.1's stream still ended after 5 m 16 s of silent thinking: “API Error: Stream ended without receiving any events”.",
"cause": "A stream that emits nothing for about 5 minutes is dropped upstream of the harness, whatever the harness settings. Fable 5.1 at max effort thinks silently for longer than that.",
"cause_html": "A stream that emits nothing for about 5 minutes is dropped upstream of the harness, whatever the harness settings. Fable 5.1 at max effort thinks silently for longer than that.",
"fix": "None on our side yet. Until the upstream limit changes, Fable 5.1 at max effort cannot finish through this setup in any harness.",
"fix_html": "None on our side yet. Until the upstream limit changes, Fable 5.1 at max effort cannot finish through this setup in any harness.",
"impact": "Claude Fable 5.1 attempt 2 on the DualSense prompt is labelled **Infrastructure DNF** (upstream silence drop): not a model failure and not a harness-config failure.",
"impact_html": "Claude Fable 5.1 attempt 2 on the DualSense prompt is labelled Infrastructure DNF (upstream silence drop): not a model failure and not a harness-config failure.",
"results": [
{
"id": "ps5_controller__claude-code__fable-5-1__attempt2",
"model": "Claude Fable 5.1",
"harness": "Claude Code 2.1.280",
"prompt_id": "ps5_controller",
"status": "infra_dnf"
}
],
"missing_results": []
},
{
"id": "opencode-stream-timeouts",
"title": "OpenCode's 5-minute stream timeouts",
"category": "harness defaults",
"status": "fixed",
"symptom": "A risk rather than an observed failure: OpenCode would abort a request that sends no chunk, or no response headers, for 5 minutes.",
"symptom_html": "A risk rather than an observed failure: OpenCode would abort a request that sends no chunk, or no response headers, for 5 minutes.",
"cause": "OpenCode defaults to a 5-minute idle-chunk timeout and a 5-minute header timeout, shorter than the silent thinking of some models at max effort.",
"cause_html": "OpenCode defaults to a 5-minute idle-chunk timeout and a 5-minute header timeout, shorter than the silent thinking of some models at max effort.",
"fix": "Raised to 30 minutes for every provider from the GPT-6 Astra OpenCode run onward, and recorded in each run's meta.",
"fix_html": "Raised to 30 minutes for every provider from the GPT-6 Astra OpenCode run onward, and recorded in each run's meta.",
"impact": "The Claude Opus 5.5 OpenCode run started before the change and used the defaults; it was not affected and finished on its own.",
"impact_html": "The Claude Opus 5.5 OpenCode run started before the change and used the defaults; it was not affected and finished on its own.",
"results": [
{
"id": "ps5_controller__opencode__opus-5-5",
"model": "Claude Opus 5.5",
"harness": "OpenCode 1.18.32",
"prompt_id": "ps5_controller",
"status": "ok"
}
],
"missing_results": []
},
{
"id": "machine-move",
"title": "The test box moved to a larger machine mid-project",
"category": "infrastructure change",
"status": "done",
"symptom": "Runs in flight when the test box was moved to a larger machine (8 GB) were cut off.",
"symptom_html": "Runs in flight when the test box was moved to a larger machine (8 GB) were cut off.",
"cause": "A planned infrastructure change, not a harness or model problem.",
"cause_html": "A planned infrastructure change, not a harness or model problem.",
"fix": "Runs cut by the move are re-run or resumed. Their cards say “restarted after machine upgrade” or “resumed after machine upgrade”, taken from the run's own record (its labels or run note).",
"fix_html": "Runs cut by the move are re-run or resumed. Their cards say “restarted after machine upgrade” or “resumed after machine upgrade”, taken from the run's own record (its labels or run note).",
"impact": "Every published run that carries one of those labels is listed with this note.",
"impact_html": "Every published run that carries one of those labels is listed with this note.",
"results": [
{
"id": "fjord_claudecode_opus55",
"model": "Claude Opus 5.5",
"harness": "Claude Code 2.1.280",
"prompt_id": "fjord",
"status": "ok"
}
],
"missing_results": []
}
],
"source": "content/field_notes.json"
},
"findings": [
{
"id": "best-controller",
"title": "Best DualSense controller so far: OpenCode with Claude Opus 5.5, at a price",
"source": "reviewer",
"html": "OpenCode + Claude Opus 5.5 (max) drew the best controller of the project so far: an accurate silhouette with an open gap between the grips, a real 4-way D-pad, correctly coloured outline symbols, knurled sticks, glowing blue light bars, a proper PS logo, a speaker grille and a small orange mute button. Its only flat spot is a plain white touchpad. It took 1h 23m and cost $18.88 (reported by OpenCode), about 1.5x the time and cost of Claude Code with the same model (56m 51s, $12.80) for a very similar look: same model, same photo-tracing approach.",
"results": [
"ps5_controller__claude-code__opus-5-5",
"ps5_controller__opencode__opus-5-5"
]
},
{
"id": "cleanest-home",
"title": "Cleanest home-harness controller, and the best value among the strong ones: Codex with GPT-6 Astra",
"source": "reviewer",
"html": "Codex CLI + GPT-6 Astra (xhigh) is the cleanest result on a home harness: an open gap between the grips and no dark slab, in a flatter, more vector look. At $2.20 (estimated) and 11m 38s it is the best result per dollar among the strong results.",
"results": [
"ps5_controller__codex__gpt-6-astra"
]
},
{
"id": "cc-opus",
"title": "Claude Code with Opus 5.5: strong materials, one layout flaw",
"source": "reviewer",
"html": "Claude Code + Claude Opus 5.5 (max) has strong materials and detail, but a dark slab fills the gap between the grips. $12.80 (reported), 56m 51s.",
"results": [
"ps5_controller__claude-code__opus-5-5"
]
},
{
"id": "dsh-value",
"title": "dsh with DeepSeek V4.1 Flash: remarkable value, clear flaws",
"source": "reviewer",
"html": "dsh + DeepSeek V4.1 Flash (max) made a bold, glossy render, but a huge dark slab fills the space between the grips, the face-button diamond is lopsided, the PS logo is garbled and the touchpad is glassy. It sits behind Codex/Astra and Claude Code/Opus. The value is remarkable: $0.19 computed cost (estimated), 19m 44s and 131 tool calls, 59 of them image views.",
"results": [
"ps5_controller__dsh__deepseek-flash"
]
},
{
"id": "fjord-codex",
"title": "Fjord scene: Codex with GPT-6 Astra is night and day better than the raw one-shot",
"source": "reviewer",
"html": "Codex CLI + GPT-6 Astra (xhigh) built green conifer slopes, snowy peaks, near-mirror reflections, a red-and-white lighthouse whose beam sweeps the water at night, a pink sunset sky and a wooden rowboat with two small rowers, in 25m 14s for about $5.57 (estimated from tokens). The raw one-shot build had flat grey cliff slabs. Weak spots: the terrain reads low-poly up close, the night stretch is very dark, the characters are tiny and simple, and rain and storm barely show. It is not at the level of the viral reference post.",
"results": [
"fjord__raw-api__opus-5-5__low",
"fjord_codex_astra"
]
},
{
"id": "setup-failures",
"title": "Three runs were lost to our setup, not to the models",
"source": "run notes",
"html": "Claude Code's stream-idle watchdog (our configuration) killed Claude Fable 5.1 attempt 1 and the first Claude Code fjord attempt with Opus 5.5, which had spent $5.03 by then. Fable 5.1 attempt 2, with the watchdog raised, was still cut upstream after about 5 minutes of silent thinking and is logged as an infrastructure DNF. None of these counts against the model; the spend is still counted. Details in the field notes.",
"results": [
"fjord_claude_code_opus55_attempt1",
"ps5_controller__claude-code__fable-5-1__attempt1",
"ps5_controller__claude-code__fable-5-1__attempt2"
]
},
{
"id": "raw-max-vs-high",
"title": "On a raw call, max effort lost to high effort",
"source": "data",
"html": "Claude Opus 5.5 at max effort spent its entire 128,000-token output budget thinking (20m 08s, $2.56) and never produced an SVG; with a 64k budget it did the same ($1.28). Effort high finished in 7m 21s for $0.91. The same happened on the animated-short prompt: max effort thought for 20m 42s and wrote nothing ($2.56), while effort low made the whole film in 1m 38s for $0.24.",
"results": [
"js_short__raw-api__opus-5-5__low",
"js_short__raw-api__opus-5-5__max",
"ps5_controller__raw-api__opus-5-5__high",
"ps5_controller__raw-api__opus-5-5__max-128k",
"ps5_controller__raw-api__opus-5-5__max-64k"
]
}
],
"featured": {
"why": "Picked from the reviewer's notes: the strongest harness results so far.",
"results": [
"ps5_controller__opencode__opus-5-5",
"ps5_controller__codex__gpt-6-astra",
"fjord_codex_astra",
"ps5_controller__claude-code__opus-5-5"
]
}
}