CODING AGENT HIGHLIGHTS · OCTOBER 2026

Verdict Code, in the numbers.

The latest parser pairs use current models chosen from the live cost, quality and speed ladders and catalog added dates. Both agents request matched sampling and reasoning settings on the unchanged task; unsupported provider settings are disclosed. Other task results are retained from the prior campaign. The highlights below feature runs where every check passed; the complete run history, with receipts, is linked further down.

Synthetic test inputs · one trial per cell · no claim of statistical significance or general product superiority.

9Completed matched-harness runs

6/6Prior native coding runs passed

36 earlier attempts, incomplete runs and diagnostics are retained below. They are excluded from coding success counts. Completed failures remain in the results.

HIGHLIGHTS

Where Verdict Code delivers.

Highlights show runs where every independent check passed. Synthetic test inputs, one trial per cell. The full run history, including runs that did not pass, is linked below.

Parser task · every check passedSynthetic parser task, 20 independent checks, current models.

Compare with Aider

Verdict Code · anthropic/claude-haiku-5-520/20 checks passed13.4 s
Verdict Code · openai/gpt-6-luna20/20 checks passed31.0 s
Verdict Code · zai/glm-5.3-flashx20/20 checks passed73.7 s

Verdict Code runs its own verification and repair loop, so it may use more tokens and time than a single call. Each bar links to its receipt in the table below.

Multi-file regression repair · budget model, full marksVerdict Code used GLM-5.3-Flash (about $0.15 per million input tokens). Native agents used their own subscription models, so models differ.

Verdict Code10/10 checks passed · zai/glm-5.3-flash10/10 checks
Devin CLI10/10 checks passed · swe-2-medium10/10 checks
Claude Code10/10 checks passed · claude-sonnet-5-510/10 checks
Codex CLI10/10 checks passed · gpt-6.1-sol10/10 checks

Blueprint planning · wall time, every check passedSynthetic specification inputs: explicit-dependency spec (13 checks) and prose spec (10 checks). Lower is faster.

Explicit dependencies

Verdict Blueprint planner13/13 checks passed · deterministic, no model call0.5 s
Devin CLI13/13 checks passed · swe-2-medium5.3 s
Codex CLI13/13 checks passed · gpt-6.1-sol17.2 s
Claude Code13/13 checks passed · claude-sonnet-5-58.8 s
Raw model planning baseline13/13 checks passed · zai/glm-5.3-flash13.9 s

Prose specification

Verdict Blueprint planner10/10 checks passed · deterministic, no model call1.0 s
Devin CLI10/10 checks passed · swe-2-medium16.5 s
Codex CLI10/10 checks passed · gpt-6.1-sol17.2 s
Claude Code10/10 checks passed · claude-sonnet-5-58.7 s
Raw model planning baseline10/10 checks passed · zai/glm-5.3-flash34.3 s

Blueprint is a Verdict feature: its planner is deterministic and calls no model, so it plans in under a second. Other agents run a model; this shows what the feature delivers, not model quality.

Results by task

Verdict Code vs Aider

Same parser task and model. Each cell shows independent checks passed. Current models are selected from live ladders and catalog added dates. Older rosters remain in the audit archive.

Synthetic benchmark fixtures · actual agent or service execution.

Exact model	Verdict Code	Aider
anthropic/claude-haiku-5-5Current model · catalog verified · synthetic parser task	Passed20/20 checks13.412 s · USD not measuredReceipt ↗ Run details Model: byok/anthropic/claude-haiku-5-5 Observed native model: Not measured Gateway response model: byok/anthropic/claude-haiku-5-5 Reasoning setting: default Input / output tokens: 11,092 / 1,655 Total tokens: 12,747 Accounted tokens / wall-second: 950.418 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Provider cost estimation returned zero despite positive usage and catalog prices. Cost is unavailable; the separately labelled catalog estimate is not an observed charge.	Passed20/20 checks14.012 s · USD not measuredReceipt ↗ Run details Model: byok/anthropic/claude-haiku-5-5 Observed native model: Not measured Gateway response model: byok/anthropic/claude-haiku-5-5 Reasoning setting: default Input / output tokens: 1,241 / 2,617 Total tokens: 3,858 Accounted tokens / wall-second: 275.335 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Provider cost estimation returned zero despite positive usage and catalog prices. Cost is unavailable; the separately labelled catalog estimate is not an observed charge.
openai/gpt-6-lunaCurrent model · catalog verified · synthetic parser task	Passed20/20 checks30.985 s · $0.001266Receipt ↗ Run details Model: byok/openai/gpt-6-luna Observed native model: Not measured Gateway response model: byok/openai/gpt-6-luna Reasoning setting: default Input / output tokens: 5,713 / 1,390 Total tokens: 7,103 Accounted tokens / wall-second: 229.24 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.	Passed20/20 checks30.003 s · $0.000872Receipt ↗ Run details Model: byok/openai/gpt-6-luna Observed native model: Not measured Gateway response model: byok/openai/gpt-6-luna Reasoning setting: default Input / output tokens: 836 / 1,576 Total tokens: 2,412 Accounted tokens / wall-second: 80.392 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
qwen/qwen3.8-flashCurrent model · catalog verified · synthetic parser task	IncompleteProcess reached the fixed time limit without completing.Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.	Passed20/20 checks220.804 s · $0.005058Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: byok/qwen/qwen3.8-flash Reasoning setting: default Input / output tokens: 934 / 10,463 Total tokens: 11,397 Accounted tokens / wall-second: 51.616 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
zai/glm-5.3-flashCurrent model · catalog verified · synthetic parser task	Failed checks18/20 checks30.955 s · $0.001061Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: low Input / output tokens: 3,984 / 927 Total tokens: 4,911 Accounted tokens / wall-second: 158.65 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.	Passed20/20 checks9.204 s · $0.000348Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: low Input / output tokens: 822 / 449 Total tokens: 1,271 Accounted tokens / wall-second: 138.092 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
zai/glm-5.3-flashxCurrent model · catalog verified · synthetic parser task	Passed20/20 checks73.742 s · $0.013988Receipt ↗ Run details Model: byok/zai/glm-5.3-flashx Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flashx Reasoning setting: default Input / output tokens: 12,279 / 7,556 Total tokens: 19,835 Accounted tokens / wall-second: 268.978 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.	Passed20/20 checks40.713 s · $0.007165Receipt ↗ Run details Model: byok/zai/glm-5.3-flashx Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flashx Reasoning setting: default Input / output tokens: 822 / 5,489 Total tokens: 6,311 Accounted tokens / wall-second: 155.012 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.

Fix a regression

Repair invoice rounding across two modules. Nine behavior checks plus one permitted-file check. Different model configurations: Verdict Code used GLM-5.3-Flash; native agents used their subscription models.

Retained from the prior campaign; not rerun here.

Synthetic benchmark fixtures · actual agent or service execution.

Agent / model	Task	Result	Time	Estimated USD	Evidence
Verdict Codezai/glm-5.3-flashBudget API model	Repair money calculations across modules (synthetic test input)Synthetic test input	Passed10/10 checks	146.71 s	$0.013120	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: See invocation receipt Input / output tokens: 76,351 / 3,337 Total tokens: 79,688 Accounted tokens / wall-second: 543.167 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Synthetic fixture, one trial, 240-second limit; not a general coding-agent ranking. 9 independent behavior unittest methods plus 1 changed-file scope check. Model and token accounting require captured provider/usage review; absent values remain unknown. Timeout/transport failure is infrastructure evidence, not a model quality failure. Complete Gateway terminal usage proof; cost is estimated BYOK USD, not a provider cash receipt.
Devin CLIswe-2-mediumSubscription model	Repair money calculations across modulesSynthetic test input	Passed10/10 checks	10.936 s	Not measured	Receipt ↗ Run details Model: swe-2-medium Observed native model: swe-2-medium Provider model receipt: Not exposed Identity source: native exporter generation_model Reasoning setting: See invocation receipt Input / output tokens: 72,162 / 1,086 Total tokens: 73,248 Accounted tokens / wall-second: 6,697.879 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.
Claude Codeclaude-sonnet-5-5Subscription model	Repair money calculations across modulesSynthetic test input	Passed10/10 checks	19.102 s	Not measured	Receipt ↗ Run details Model: claude-sonnet-5-5 Observed native model: claude-sonnet-5-5 Provider model receipt: Not exposed Identity source: native exporter modelUsage canonical model Reasoning setting: See invocation receipt Input / output tokens: 129,952 / 1,655 Total tokens: 131,607 Accounted tokens / wall-second: 6,889.697 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.
Codex CLIgpt-6.1-solSubscription model	Repair money calculations across modulesSynthetic test input	Passed10/10 checks	37.556 s	Not measured	Receipt ↗ Run details Model: gpt-6.1-sol Observed native model: gpt-6.1-sol Provider model receipt: Not exposed Identity source: native-effective-config; provider response identity not exposed Reasoning setting: See invocation receipt Input / output tokens: 114,952 / 1,175 Total tokens: 116,127 Accounted tokens / wall-second: 3,092.102 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion. Provider-response model identity not exposed; native effective selection is recorded separately and does not change acceptance status.

Follow the repository rules

Implement a filename utility with the standard library, preserving the sentinel and permitted change surface.

Retained from the prior campaign; not rerun here.

Synthetic benchmark fixtures · actual agent or service execution.

Agent / model	Task	Result	Time	Estimated USD	Evidence
Verdict Codezai/glm-5.3-flashBudget API model	Add a safe filename utility without dependency or scope drift (synthetic test input)Synthetic test input	Passed10/10 checks	30.96 s	$0.001267	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: See invocation receipt Input / output tokens: 7,064 / 414 Total tokens: 7,478 Accounted tokens / wall-second: 241.537 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Synthetic fixture, one trial, 240-second limit; not a general coding-agent ranking. 9 independent behavior unittest methods plus 1 changed-file scope check. Model and token accounting require captured provider/usage review; absent values remain unknown. Timeout/transport failure is infrastructure evidence, not a model quality failure. Protocol update: public --task-file filenames.py --pin-task-files --no-ide metadata explicitly scopes context to the same one-file contract already stated in the unchanged prompt. Earlier full-context attempts remain retained; model/task/oracle are unchanged. Complete Gateway terminal usage proof; cost is estimated BYOK USD, not a provider cash receipt.
Devin CLIswe-2-mediumSubscription model	Add a safe filename utility without dependency or scope driftSynthetic test input	Passed10/10 checks	7.391 s	Not measured	Receipt ↗ Run details Model: swe-2-medium Observed native model: swe-2-medium Provider model receipt: Not exposed Identity source: native exporter generation_model Reasoning setting: See invocation receipt Input / output tokens: 69,917 / 563 Total tokens: 70,480 Accounted tokens / wall-second: 9,535.922 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.
Codex CLIgpt-6.1-solSubscription model	Add a safe filename utility without dependency or scope driftSynthetic test input	Passed10/10 checks	25.571 s	Not measured	Receipt ↗ Run details Model: gpt-6.1-sol Observed native model: gpt-6.1-sol Provider model receipt: Not exposed Identity source: native-effective-config; provider response identity not exposed Reasoning setting: See invocation receipt Input / output tokens: 91,480 / 710 Total tokens: 92,190 Accounted tokens / wall-second: 3,605.256 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion. Provider-response model identity not exposed; native effective selection is recorded separately and does not change acceptance status.
Claude Codeclaude-sonnet-5-5Subscription model	Add a safe filename utility without dependency or scope driftSynthetic test input	Passed10/10 checks	16.218 s	Not measured	Receipt ↗ Run details Model: claude-sonnet-5-5 Observed native model: claude-sonnet-5-5 Provider model receipt: Not exposed Identity source: native exporter modelUsage canonical model Reasoning setting: See invocation receipt Input / output tokens: 85,576 / 908 Total tokens: 86,484 Accounted tokens / wall-second: 5,332.593 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion. Original raw receipt retained FAIL9/10 because pytest runtime cache was counted as authored code. Independent replay corrected this oracle inconsistency: same candidate9/9 behavior plus source scope passes. No new inference or response replacement.

Blueprint planning

Two frozen specifications given to native coding agents. Scope coverage, exclusions and dependency structure are checked separately from application execution. The Verdict Blueprint planner is deterministic and makes no model call; its row shows what the feature does, not model quality. A raw-model planning baseline (GLM-5.3-Flash) is shown for reference.

Retained from the prior campaign; not rerun here.

Synthetic benchmark fixtures · actual agent or service execution.

Agent / model	Explicit dependencies	Prose specification
Verdict Blueprint plannerDeterministic, no model callSynthetic specification inputs	Passed13/13 checks0.455 sReceipt ↗ Run details Model: No inference Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: 0 / 0 Total tokens: 0 Accounted tokens / wall-second: 0 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Synthetic PRD input; planning-only: no generated-file, build, or code-execution proof. One trial per task and arm; exact rubric is deliberately narrow; no statistical or production claim. Heuristic planning, no model invocation; zero tokens are actual estimation inference usage. Forecast tokens/cost/time are excluded from actual run metrics. Energy unmeasured. Semantic rubric v2 excludes exact task count from acceptance. Count is diagnostic only; exclusion mentions are evaluated for explicit delivery versus reaffirmed non-delivery. Ambiguous wording remains UNVERIFIED. Semantic-v2 rerun uses originalunchanged synthetic PRDs. Originalv1 attempts remain published separately. No generated-code execution claim.	Passed10/10 checks0.958 sReceipt ↗ Run details Model: No inference Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: 0 / 0 Total tokens: 0 Accounted tokens / wall-second: 0 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Synthetic PRD input; planning-only: no generated-file, build, or code-execution proof. One trial per task and arm; exact rubric is deliberately narrow; no statistical or production claim. Heuristic planning, no model invocation; zero tokens are actual estimation inference usage. Forecast tokens/cost/time are excluded from actual run metrics. Energy unmeasured. Semantic rubric v2 excludes exact task count from acceptance. Count is diagnostic only; exclusion mentions are evaluated for explicit delivery versus reaffirmed non-delivery. Ambiguous wording remains UNVERIFIED. Semantic-v2 rerun uses originalunchanged synthetic PRDs. Originalv1 attempts remain published separately. No generated-code execution claim.
Devin CLIswe-2-mediumSynthetic specification inputs	Passed13/13 checks5.329 sReceipt ↗ Run details Model: swe-2-medium Observed native model: swe-2-medium Provider model receipt: Not exposed Identity source: native exporter generation_model Reasoning setting: See invocation receipt Input / output tokens: 27,466 / 512 Total tokens: 27,978 Accounted tokens / wall-second: 5,250.141 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Planning structure rubric only; exact task count and exclusion text checks require semantic review. Not code execution quality. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.	Passed10/10 checks16.52 sReceipt ↗ Run details Model: swe-2-medium Observed native model: swe-2-medium Provider model receipt: Not exposed Identity source: native exporter generation_model Reasoning setting: See invocation receipt Input / output tokens: 27,503 / 781 Total tokens: 28,284 Accounted tokens / wall-second: 1,712.107 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Planning structure rubric only; exact task count and exclusion text checks require semantic review. Not code execution quality. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.
Codex CLIgpt-6.1-solSynthetic specification inputs	Passed13/13 checks17.165 sReceipt ↗ Run details Model: gpt-6.1-sol Observed native model: gpt-6.1-sol Provider model receipt: Not exposed Identity source: native-effective-config; provider response identity not exposed Reasoning setting: See invocation receipt Input / output tokens: 48,367 / 537 Total tokens: 48,904 Accounted tokens / wall-second: 2,849.053 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Planning structure rubric only; exact task count and exclusion text checks require semantic review. Not code execution quality. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion. Provider-response model identity not exposed; native effective selection is recorded separately and does not change acceptance status.	Passed10/10 checks17.226 sReceipt ↗ Run details Model: gpt-6.1-sol Observed native model: gpt-6.1-sol Provider model receipt: Not exposed Identity source: native-effective-config; provider response identity not exposed Reasoning setting: See invocation receipt Input / output tokens: 48,800 / 530 Total tokens: 49,330 Accounted tokens / wall-second: 2,863.694 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Planning structure rubric only; exact task count and exclusion text checks require semantic review. Not code execution quality. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion. Provider-response model identity not exposed; native effective selection is recorded separately and does not change acceptance status.
Claude Codeclaude-sonnet-5-5Synthetic specification inputs	Passed13/13 checks8.83 sReceipt ↗ Run details Model: claude-sonnet-5-5 Observed native model: claude-sonnet-5-5 Provider model receipt: Not exposed Identity source: native exporter modelUsage canonical model Reasoning setting: See invocation receipt Input / output tokens: 42,329 / 849 Total tokens: 43,178 Accounted tokens / wall-second: 4,889.921 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Planning structure rubric only; exact task count and exclusion text checks require semantic review. Not code execution quality. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.	Passed10/10 checks8.677 sReceipt ↗ Run details Model: claude-sonnet-5-5 Observed native model: claude-sonnet-5-5 Provider model receipt: Not exposed Identity source: native exporter modelUsage canonical model Reasoning setting: See invocation receipt Input / output tokens: 42,355 / 753 Total tokens: 43,108 Accounted tokens / wall-second: 4,968.077 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Planning structure rubric only; exact task count and exclusion text checks require semantic review. Not code execution quality. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.
Raw model planning baselinezai/glm-5.3-flashSynthetic specification inputs	Passed13/13 checks13.895 sReceipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Synthetic PRD input; planning-only: no generated-file, build, or code-execution proof. One trial per task and arm; exact rubric is deliberately narrow; no statistical or production claim. Raw model baseline, not a competing coding agent. Requested model held constant only within this baseline; Blueprint has no inference model. Cost is published-rate estimate at $0.15 input/$0.50 output per MTok; provider cash receipt not captured. Semantic rubric v2 excludes exact task count from acceptance. Count is diagnostic only; exclusion mentions are evaluated for explicit delivery versus reaffirmed non-delivery. Ambiguous wording remains UNVERIFIED. Semantic-v2 rerun uses originalunchanged synthetic PRDs. Originalv1 attempts remain published separately. No generated-code execution claim.	Passed10/10 checks34.303 sReceipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Synthetic PRD input; planning-only: no generated-file, build, or code-execution proof. One trial per task and arm; exact rubric is deliberately narrow; no statistical or production claim. Raw model baseline, not a competing coding agent. Requested model held constant only within this baseline; Blueprint has no inference model. Cost is published-rate estimate at $0.15 input/$0.50 output per MTok; provider cash receipt not captured. Semantic rubric v2 excludes exact task count from acceptance. Count is diagnostic only; exclusion mentions are evaluated for explicit delivery versus reaffirmed non-delivery. Ambiguous wording remains UNVERIFIED. Semantic-v2 rerun uses originalunchanged synthetic PRDs. Originalv1 attempts remain published separately. No generated-code execution claim.

Local models: Verdict Code vs Aider (same visible test)

Free LAN LM Studio route, no currency spend: gpt-oss-120b and qwen3-coder-30b, Verdict Code 1.4.600 and Aider. Both agents get the same visible test (python3 test_visible.py). A hidden scorer with 20 assertions decides a pass. Verdict Code ran 6 trials per model (3 at default temperature, 3 at 0.0); Aider ran 3.

Synthetic test input (frozen Retry-After task) · actual CLIs · free LAN route. Session-1595.

Model	Verdict Code (all 6 trials)	Aider (3 trials)	Like-for-like at default temperature (Verdict Code vs Aider)
gpt-oss-120b	6/6 passed 20/20	3/3 passed 20/20	3/3 vs 3/3 (tie)
qwen3-coder-30b	4/6 passed 20/20	0/3 passed 20/20	2/3 vs 0/3

Honest notes.

The gpt-oss-120b result is a tie, not a win. At default temperature both agents passed 3/3.
Every qwen3-coder-30b failure on both sides was the same non-ASCII-digit case. All three Aider trials scored 18/20.
When Verdict Code's visible test was still failing after 3 repair turns, it reported incomplete instead of claiming done. Both qwen non-passes (17/20 and 18/20) ended this way.
Aider's token counts repeat across trials 2 and 3 (deterministic sampling).
One gpt-oss Verdict Code cell (20/20) is excluded because the Gateway restarted after it; it was rerun. 18 eligible cells are listed below.
Small sample, not a significance test; this does not show general product superiority.

Per-cell calls, tokens and wall time (18 eligible cells)

Model	Arm	Trial	Result	Calls	Total tokens	Wall (s)
gpt-oss-120b	Verdict Code, default temp (trials 1-3)	1	Passed · 20/20	1	3,312	26.1
gpt-oss-120b	Verdict Code, default temp (trials 1-3)	2	Passed · 20/20	1	3,464	27.6
gpt-oss-120b	Verdict Code, default temp (trials 1-3)	3	Passed · 20/20	4	9,307	44.7
gpt-oss-120b	Verdict Code, temp 0.0 (trials 4-6)	4	Passed · 20/20	2	6,413	40.7
gpt-oss-120b	Verdict Code, temp 0.0 (trials 4-6)	5	Passed · 20/20	4	8,705	39.4
gpt-oss-120b	Verdict Code, temp 0.0 (trials 4-6)	6	Passed · 20/20	2	5,610	31.5
gpt-oss-120b	Aider, 3 trials	1	Passed · 20/20	2	7,579	51.5
gpt-oss-120b	Aider, 3 trials	2	Passed · 20/20	2	6,235	44.6
gpt-oss-120b	Aider, 3 trials	3	Passed · 20/20	2	6,235	44.6
qwen3-coder-30b	Verdict Code, default temp (trials 1-3)	1	Passed · 20/20	3	11,308	28.5
qwen3-coder-30b	Verdict Code, default temp (trials 1-3)	2	Incomplete · 17/20	4	14,556	23.5
qwen3-coder-30b	Verdict Code, default temp (trials 1-3)	3	Passed · 20/20	2	6,932	18.9
qwen3-coder-30b	Verdict Code, temp 0.0 (trials 4-6)	4	Incomplete · 18/20	4	13,663	21.4
qwen3-coder-30b	Verdict Code, temp 0.0 (trials 4-6)	5	Passed · 20/20	4	16,003	35.4
qwen3-coder-30b	Verdict Code, temp 0.0 (trials 4-6)	6	Passed · 20/20	3	10,381	17.8
qwen3-coder-30b	Aider, 3 trials	1	Failed checks · 18/20	4	10,208	27.6
qwen3-coder-30b	Aider, 3 trials	2	Failed checks · 18/20	4	10,279	26.5
qwen3-coder-30b	Aider, 3 trials	3	Failed checks · 18/20	4	10,279	26.3

Receipts: Per-cell reviewed receipts ↗ · Spend summary (estimated USD 0.00, free LAN route) ↗

Same model, different agents

The comparisons above give each agent its own model. Here the model is held fixed, so differences come from the agent. Two frozen coding tasks, the same hidden checks (9 behaviour tests and 1 changed-file scope check), 3 runs per agent per task. Verdict Code (rerun on 1.4.618, October 20) pins the model through the Gateway; Devin, Claude Code and Codex use their own CLIs with the same requested model.

Synthetic test input (frozen coding fixtures) · actual agent CLIs · October 18–20, 2026.

Claude Sonnet 5.5: mean wall time, every run 10/10

Task	Verdict Code	Devin CLI	Claude Code
Money bug across 2 files	17.8 s (15.3–19.8, n=3) · 4.0 calls · was 19.1 s, 7.0 calls on 1.4.603	12.0 s (10.3–13.1, n=3) · 4.0 calls	26.4 s (18.9–30.6, n=3) · 7.0 calls
Safe filename utility	10.2 s (9.7–10.8, n=3) · 2.0 calls · was 6.5 s, 2.0 calls on 1.4.603	8.9 s (7.3–11.1, n=3) · 3.3 calls	16.1 s (10.8–19.1, n=3) · 4.7 calls

GPT-6.1 Sol: mean wall time, every run 10/10

Task	Verdict Code	Devin CLI	Codex CLI
Money bug across 2 files	46.0 s (44.0–47.7, n=3) · 6.0 calls · was 45.9 s, 14.0 calls on 1.4.603	39.8 s (38.4–42.0, n=3) · 5.3 calls	31.5 s (26.9–36.8, n=3) · calls not exposed
Safe filename utility	14.8 s (13.3–16.3, n=3) · 2.7 calls · was 14.6 s, 3.0 calls on 1.4.603	21.2 s (19.9–22.2, n=3) · 5.0 calls	20.0 s (15.3–22.6, n=3) · calls not exposed

Honest notes.

Verdict Code 1.4.618 cut its money-bug calls from 14 to 6 on GPT-6.1 Sol and from 7 to 4 on Sonnet 5.5, mainly by reading several files in one call. Devin makes 5.3 and 4.
Wall time did not fall with the calls: on GPT-6.1 Sol the money bug still takes 46.0 s, against 39.8 s for Devin and 31.5 s for Codex. On Sonnet 5.5 Devin is faster on both tasks; on GPT-6.1 Sol Verdict Code stays fastest on the filename task.
Devin, Claude Code and Codex were not rerun; their rows are from October 18–19.
Effort tiers are not matched: Claude Code and Codex ran at low, Devin at "medium" (part of its model name), Verdict Code at the model default.
Codex does not expose the model that served the request; only the requested gpt-6.1-sol is known.
4 infrastructure rows (a Gateway tool-choice rejection, a completion-guard defect fixed in 1.4.603, an expired Claude Code login, and a Gateway 401 during a Verdict Server restart on the 1.4.618 retest) are kept in the receipts and excluded from the tables.
3 runs per cell on two synthetic tasks: no significance test and no ranking claim.

Receipts: Per-run receipts (52 rows) ↗ · Spend summary (about USD 0.73 counted, plus USD 0.40 for the 1.4.618 retest) ↗

WHAT CHANGED

The evidence, explained.

The current comparison uses five exact inexpensive models chosen from the live cost, quality and speed ladders and catalog added dates. Earlier model rosters remain only in audit history.
Verdict Code passed 20/20 on Haiku 5.5, GPT-6 Luna and GLM-5.3-FlashX, and added no measurable latency on Haiku 5.5 (13.412 s versus 14.012 s for Aider). GPT-6 Luna passed 20/20 in 30.985 s and 30.003 s respectively. These are single observations on one synthetic parser task.
GLM-5.3-FlashX passed both arms, but Verdict Code took 73.742 s versus 40.713 s. The latest GLM-5.3-Flash run finished in 30.955 s but failed 2 of 20 checks, versus Aider passing 20/20 in 9.204 s.
Qwen 3.8 Flash remains incomplete at the original four-minute limit, despite a partial artifact passing 20/20. Aider completed in 220.804 s. Earlier timeout and stale-credential interruption remain retained; a correct partial artifact is not a completed pass.
Generic repairs address redundant provider turns, cumulative repair bounds, stale self-check evidence, lost workspace locations, contaminated check imports, and required Responses compatibility. Hidden checks were not supplied to candidate execution or used to tune repairs.
All original publications remain byte-preserved. Other native-agent, Blueprint and coding-task families are retained prior observations, not fresh reruns. Incomplete accounting stays unknown; these results do not establish a general agent ranking.

Full audit history

Earlier attempts and execution problems remain visible. They are separate from completed coding results.

Earlier attempts, incomplete runs and diagnostics (36)

Agent / model	Task	Result	Time	Estimated USD	Evidence
Verdict Codeopenai/gpt-4.1-mini	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete18/20 checks on partial artifactEarlier attempt; superseded by a recorded later runHistorical roster · retained for auditIncomplete or apparatus diagnosticblocked · see receipt for cause	57.415 s	Not measured	Receipt ↗ Run details Model: byok/openai/gpt-4.1-mini Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. A metadata probe disconnected; inference is assessed separately from captured POST completion and accounting. Gateway instance continuity was not proven; excluded from matched-model comparison. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
Verdict Codeopenai/gpt-4.1-mini	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete15/20 checks on partial artifactEarlier attempt; superseded by a recorded later runHistorical roster · retained for auditIncomplete or apparatus diagnosticblocked · see receipt for cause	75.334 s	$0.022086	Receipt ↗ Run details Model: byok/openai/gpt-4.1-mini Observed native model: Not measured Gateway response model: byok/openai/gpt-4.1-mini Reasoning setting: default Input / output tokens: 27,387 / 6,957 Total tokens: 34,344 Accounted tokens / wall-second: 455.89 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
Verdict Codedeepseek/deepseek-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksHistorical roster · retained for auditIncomplete or apparatus diagnostic	18.714 s	$0.003283	Receipt ↗ Run details Model: byok/deepseek/deepseek-flash Observed native model: Not measured Gateway response model: byok/deepseek/deepseek-flash Reasoning setting: default Input / output tokens: 1,253 / 2,423 Total tokens: 3,676 Accounted tokens / wall-second: 196.43 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
Verdict Codeopenai/gpt-4.1-mini	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks18/20 checksHistorical roster · retained for auditIncomplete or apparatus diagnostic	11.536 s	$0.001131	Receipt ↗ Run details Model: byok/openai/gpt-4.1-mini Observed native model: Not measured Gateway response model: byok/openai/gpt-4.1-mini Reasoning setting: default Input / output tokens: 915 / 478 Total tokens: 1,393 Accounted tokens / wall-second: 120.752 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
Verdict Codezai/glm-5.3-flashBudget API model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksEarlier attempt; superseded by a recorded later runIncomplete or apparatus diagnostic	23.55 s	$0.000378	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: low Input / output tokens: 1,102 / 425 Total tokens: 1,527 Accounted tokens / wall-second: 64.841 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
Verdict Codeqwen/qwen3-coder-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks16/20 checksHistorical roster · retained for auditIncomplete or apparatus diagnostic	19.527 s	$0.002548	Receipt ↗ Run details Model: byok/qwen/qwen3-coder-flash Observed native model: Not measured Gateway response model: byok/qwen/qwen3-coder-flash Reasoning setting: default Input / output tokens: 3,203 / 1,058 Total tokens: 4,261 Accounted tokens / wall-second: 218.211 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
Verdict Codegoogle/models/gemini-2.5-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks16/20 checksHistorical roster · retained for auditIncomplete or apparatus diagnostic	19.476 s	$0.007063	Receipt ↗ Run details Model: byok/google/models/gemini-2.5-flash Observed native model: Not measured Gateway response model: byok/google/models/gemini-2.5-flash Reasoning setting: default Input / output tokens: 992 / 2,706 Total tokens: 3,698 Accounted tokens / wall-second: 189.875 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
Verdict Codeqwen/qwen3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksEarlier attempt; superseded by a recorded later runIncomplete or apparatus diagnostic	59.14 s	$0.001914	Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: byok/qwen/qwen3.8-flash Reasoning setting: default Input / output tokens: 1,280 / 3,664 Total tokens: 4,944 Accounted tokens / wall-second: 83.598 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
Verdict Codegoogle/models/gemini-3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	IncompleteHistorical roster · retained for auditIncomplete or apparatus diagnosticblocked · see receipt for cause	Not measured	Not measured	Receipt ↗ Run details Model: byok/google/models/gemini-3.8-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Interrupted old-roster attempt. No terminal completion, wall measurement or independent scoring claimed. Captured token subtotals, if any, are lower bounds only. Provider output reached the fixed per-request ceiling; censored attempt, not a task-quality failure. Gateway instance continuity was not proven; excluded from matched-model comparison. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
Verdict Codeanthropic/claude-haiku-5-5	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksEarlier attempt; superseded by a recorded later run	205.045 s	Not measured	Receipt ↗ Run details Model: byok/anthropic/claude-haiku-5-5 Observed native model: Not measured Gateway response model: byok/anthropic/claude-haiku-5-5 Reasoning setting: default Input / output tokens: 24,556 / 4,490 Total tokens: 29,046 Accounted tokens / wall-second: 141.657 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Provider cost estimation returned zero despite positive usage and catalog prices. Cost is unavailable; the separately labelled catalog estimate is not an observed charge.
Verdict Codeopenai/gpt-6-luna	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete0/20 checks on partial artifactEarlier attempt; superseded by a recorded later runIncomplete or apparatus diagnosticblocked · see receipt for cause	3.788 s	Not measured	Receipt ↗ Run details Model: byok/openai/gpt-6-luna Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
Verdict Codeqwen/qwen3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete20/20 checks on partial artifactEarlier attempt; superseded by a recorded later runIncomplete or apparatus diagnostictimeout · see receipt for cause	240.32 s	Not measured	Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
Verdict Codezai/glm-5.3-flashBudget API model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksEarlier attempt; superseded by a recorded later run	84.191 s	$0.004326	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: low Input / output tokens: 20,417 / 2,526 Total tokens: 22,943 Accounted tokens / wall-second: 272.511 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
Verdict Codeqwen/qwen3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete20/20 checks on partial artifactEarlier attempt; superseded by a recorded later runIncomplete or apparatus diagnosticblocked · see receipt for cause	153.084 s	Not measured	Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
Verdict Codeqwen/qwen3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete20/20 checks on partial artifactIncomplete or apparatus diagnostictimeout · see receipt for cause	240.628 s	Not measured	Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
Verdict Codezai/glm-5.3-flashBudget API model	Repair money calculations across modules (synthetic test input)Synthetic test input	IncompleteIncomplete or apparatus diagnosticblocked · see receipt for cause	Not measured	Not measured	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Gateway health preflight raised ConnectError [Errno61] Connection refused before agent/provider invocation. No coding quality result or spend inferred. This preflight diagnostic was reconstructed immediately after wrapper failure; elapsed time was not durably captured and remains unknown.
Verdict Codezai/glm-5.3-flashBudget API model	Add a safe filename utility without dependency or scope drift (synthetic test input)Synthetic test input	IncompleteIncomplete or apparatus diagnosticblocked · see receipt for cause	4.061 s	Not measured	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured CLI rejected a non-private 0755 instrumentation directory before inference. Corrected runner uses 0700. Configuration diagnostic, not a coding failure. Synthetic fixture, one trial, 240-second limit; not a general coding-agent ranking. 9 independent behavior unittest methods plus 1 changed-file scope check. Model and token accounting require captured provider/usage review; absent values remain unknown. Timeout/transport failure is infrastructure evidence, not a model quality failure.
Verdict Codezai/glm-5.3-flashBudget API model	Add a safe filename utility without dependency or scope drift (synthetic test input)Synthetic test input	Incomplete10/10 checks on partial artifactblocked · see receipt for cause	56.813 s	$0.004217	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: See invocation receipt Input / output tokens: 25,815 / 691 Total tokens: 26,506 Accounted tokens / wall-second: 466.548 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Gateway lifecycle persistence failed for request 19908254-ebcc-4ad1-867e-7dfa65664468: terminal durability 500 ms deadline expired beside a 996 ms Gateway loop stall. No timeout or token-budget failure. Complete usage accounting survived. Generated code independently passes 9 behavior tests plus 1 scope check, but the CLI did not terminate successfully. Do not count as completed product PASS. Synthetic fixture, one trial, 240-second limit; not a general coding-agent ranking. 9 independent behavior unittest methods plus 1 changed-file scope check. Model and token accounting require captured provider/usage review; absent values remain unknown. Timeout/transport failure is infrastructure evidence, not a model quality failure. Complete Gateway terminal usage proof; cost is estimated BYOK USD, not a provider cash receipt.
Verdict Codezai/glm-5.3-flashBudget API model	Repair money calculations across modules (synthetic test input)Synthetic test input	Passed10/10 checksIncomplete or apparatus diagnostic	156.373 s	$0.014683	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: See invocation receipt Input / output tokens: 83,930 / 4,190 Total tokens: 88,120 Accounted tokens / wall-second: 563.524 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Instrumentation diagnostic: candidate read provider traces inside workspace. Completion/code checks are real; timing and token usage are confounded. A mistaken interrupted reconstruction was corrected from original terminal stdout/journal; correction retained. Synthetic fixture, one trial, 240-second limit; not a general coding-agent ranking. 9 independent behavior unittest methods plus 1 changed-file scope check. Model and token accounting require captured provider/usage review; absent values remain unknown. Timeout/transport failure is infrastructure evidence, not a model quality failure. Complete Gateway terminal usage proof; cost is estimated BYOK USD, not a provider cash receipt.
Verdict Codezai/glm-5.3-flashBudget API model	Repair money calculations across modules (synthetic test input)Synthetic test input	IncompleteIncomplete or apparatus diagnosticblocked · see receipt for cause	Not measured	Not measured	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Gateway health preflight raised ConnectError [Errno61] Connection refused before agent/provider invocation. No coding quality result or spend inferred. This preflight diagnostic was reconstructed immediately after wrapper failure; elapsed time was not durably captured and remains unknown.
Verdict Codezai/glm-5.3-flashBudget API model	Repair money calculations across modules (synthetic test input)Synthetic test input	IncompleteIncomplete or apparatus diagnosticblocked · see receipt for cause	3.768 s	Not measured	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured CLI rejected a non-private 0755 instrumentation directory before inference. Corrected runner uses 0700. Configuration diagnostic, not a coding failure. Synthetic fixture, one trial, 240-second limit; not a general coding-agent ranking. 9 independent behavior unittest methods plus 1 changed-file scope check. Model and token accounting require captured provider/usage review; absent values remain unknown. Timeout/transport failure is infrastructure evidence, not a model quality failure.
aideropenai/gpt-4.1-mini	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks18/20 checksHistorical roster · retained for audit	10.293 s	$0.001265	Receipt ↗ Run details Model: byok/openai/gpt-4.1-mini Observed native model: Not measured Gateway response model: byok/openai/gpt-4.1-mini Reasoning setting: default Input / output tokens: 831 / 583 Total tokens: 1,414 Accounted tokens / wall-second: 137.375 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aiderdeepseek/deepseek-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete0/20 checks on partial artifactHistorical roster · retained for auditIncomplete or apparatus diagnosticblocked · see receipt for cause	48.921 s	$0.010084	Receipt ↗ Run details Model: byok/deepseek/deepseek-flash Observed native model: Not measured Gateway response model: byok/deepseek/deepseek-flash Reasoning setting: default Input / output tokens: 847 / 8,192 Total tokens: 9,039 Accounted tokens / wall-second: 184.767 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Provider output reached the fixed per-request ceiling; censored attempt, not a task-quality failure. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aiderqwen/qwen3-coder-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks16/20 checksHistorical roster · retained for audit	11.421 s	$0.001048	Receipt ↗ Run details Model: byok/qwen/qwen3-coder-flash Observed native model: Not measured Gateway response model: byok/qwen/qwen3-coder-flash Reasoning setting: default Input / output tokens: 847 / 529 Total tokens: 1,376 Accounted tokens / wall-second: 120.48 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aiderzai/glm-5.3-flashBudget API model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksEarlier attempt; superseded by a recorded later run	8.01 s	$0.000332	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: low Input / output tokens: 822 / 417 Total tokens: 1,239 Accounted tokens / wall-second: 154.682 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aidergoogle/models/gemini-2.5-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks16/20 checksHistorical roster · retained for audit	18.183 s	$0.008781	Receipt ↗ Run details Model: byok/google/models/gemini-2.5-flash Observed native model: Not measured Gateway response model: byok/google/models/gemini-2.5-flash Reasoning setting: default Input / output tokens: 870 / 3,408 Total tokens: 4,278 Accounted tokens / wall-second: 235.275 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aiderqwen/qwen3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete0/20 checks on partial artifactEarlier attempt; superseded by a recorded later runIncomplete or apparatus diagnostictimeout · see receipt for cause	240.012 s	Not measured	Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Gateway instance continuity was not proven; excluded from matched-model comparison. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
aidergoogle/models/gemini-3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete0/20 checks on partial artifactHistorical roster · retained for auditIncomplete or apparatus diagnosticblocked · see receipt for cause	55.042 s	$0.031358	Receipt ↗ Run details Model: byok/google/models/gemini-3.8-flash Observed native model: Not measured Gateway response model: byok/google/models/gemini-3.8-flash Reasoning setting: default Input / output tokens: 870 / 8,188 Total tokens: 9,058 Accounted tokens / wall-second: 164.565 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Provider output reached the fixed per-request ceiling; censored attempt, not a task-quality failure. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aiderverdict/verdict-4-35b-toolterse-v5-mlxLocal/LAN model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks18/20 checksHistorical roster · retained for audit	7.141 s	$0.000000	Receipt ↗ Run details Model: local/lmstudio/verdict/verdict-4-35b-toolterse-v5-mlx Observed native model: Not measured Gateway response model: local/lmstudio/verdict/verdict-4-35b-toolterse-v5-mlx Reasoning setting: default Input / output tokens: 886 / 438 Total tokens: 1,324 Accounted tokens / wall-second: 185.408 (includes input and cached tokens; not decode speed) Energy estimate: 0.257 Wh Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aiderqwen/qwen3-coder-30bLocal/LAN model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks18/20 checksHistorical roster · retained for audit	8.596 s	$0.000000	Receipt ↗ Run details Model: local/lmstudio/qwen/qwen3-coder-30b Observed native model: Not measured Gateway response model: local/lmstudio/qwen/qwen3-coder-30b Reasoning setting: default Input / output tokens: 847 / 574 Total tokens: 1,421 Accounted tokens / wall-second: 165.309 (includes input and cached tokens; not decode speed) Energy estimate: 0.276 Wh Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aideropenai/gpt-oss-20bLocal/LAN model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks16/20 checksHistorical roster · retained for audit	25.471 s	$0.000000	Receipt ↗ Run details Model: local/lmstudio/openai/gpt-oss-20b Observed native model: Not measured Gateway response model: local/lmstudio/openai/gpt-oss-20b Reasoning setting: default Input / output tokens: 900 / 340 Total tokens: 1,240 Accounted tokens / wall-second: 48.683 (includes input and cached tokens; not decode speed) Energy estimate: 0.241 Wh Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aidergoogle/gemma-4-26b-a4bLocal/LAN model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete0/20 checks on partial artifactHistorical roster · retained for auditIncomplete or apparatus diagnostictimeout · see receipt for cause	240.091 s	Not measured	Receipt ↗ Run details Model: local/lmstudio/google/gemma-4-26b-a4b Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
aiderqwen/qwen3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksEarlier attempt; superseded by a recorded later run	192.155 s	$0.005245	Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: byok/qwen/qwen3.8-flash Reasoning setting: default Input / output tokens: 934 / 10,861 Total tokens: 11,795 Accounted tokens / wall-second: 61.383 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aiderqwen/qwen2.5-coder-1.5b-instructLocal/LAN model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	IncompleteHistorical roster · retained for auditIncomplete or apparatus diagnosticblocked · see receipt for cause	Not measured	Not measured	Receipt ↗ Run details Model: local/lmstudio/qwen/qwen2.5-coder-1.5b-instruct Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Interrupted old-roster attempt. No terminal completion, wall measurement or independent scoring claimed. Captured token subtotals, if any, are lower bounds only. Gateway instance continuity was not proven; excluded from matched-model comparison. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
aideropenai/gpt-6-luna	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksEarlier attempt; superseded by a recorded later run	21.078 s	$0.000832	Receipt ↗ Run details Model: byok/openai/gpt-6-luna Observed native model: Not measured Gateway response model: byok/openai/gpt-6-luna Reasoning setting: default Input / output tokens: 836 / 1,497 Total tokens: 2,333 Accounted tokens / wall-second: 110.684 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
AntigravityDeterministic, no model callSubscription model	Repeat isolated native execution access preflightSynthetic test input	IncompleteIncomplete or apparatus diagnosticblocked · see receipt for cause	Not measured	Not measured	Receipt ↗ Run details Model: No inference Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Access preflight only; no model invocation and no coding failure. Installed CLI advertises editor operations, with no isolated headless coding-agent command. Native UI conversations were not disturbed. No measured inference wall time, tokens, billing or quality result.

Previous harness repair — all 30 new attempts preserved unchanged

The preceding parser campaign and its retained task families remain available with their original receipts and conclusions.

Previous harness result snapshot ↗ · Previous harness receipt manifest ↗

First corrected campaign — 58 records preserved unchanged

This earlier campaign used different sampling settings. Source fixes and settings both changed; scores are not pooled.

Prior result snapshot ↗ · Prior receipt manifest ↗

Original October 15 pilot — preserved unchanged

The original 57 records include the first runner setup, short time limits, interruptions and incomplete accounting. Reruns use a revised protocol and are not pooled with those results.

Original result snapshot ↗ · Original 57-receipt manifest ↗

How to read this

Passed means the independent task checks passed and the agent finished. Missing model identity or billing measurements remain unknown; they do not turn correct code into a failure. Incomplete runs do not establish a coding result.

Native agents use their own subscription models. The paired table holds the requested model constant; recorded settings, shared hardware and run order still apply. USD figures are estimates, and subscription estimates are not settled charges.

Download evidence manifest ↗

Current result snapshot ↗ · Model selection and added dates ↗

Unchanged frozen parser prompt, starter and hidden oracle; both actual CLIs. The user replaced the obsolete roster with five current inexpensive models, selected from live ladders and catalog added dates; historical runs remain in audit history. Every historical campaign and every new attempt remains available.
Both agents request temperature zero and retain fixed 240-second, per-request and cumulative token limits. GPT-6 Luna uses Responses with the same provider-default reasoning in both arms; its unsupported custom sampling is omitted upstream, as required by the vendor. No silent model substitution or hidden-check feedback.
Generic product changes remove redundant read and completion turns, bound repeated self-repair and provide an explicit text tool protocol for models without native calls. Protocol differences are disclosed in receipts.
The independent hidden scorer runs only after candidate execution. It supplies no examples, diagnostics or expected outputs to candidates or repair decisions.
The results tables show the latest scheduled attempt per cell with no best-of selection; the Highlights section features fully passing runs only. Causal apparatus retries remain in history.
Other coding and Blueprint families retain prior measured receipts; they were not rerun in this parser campaign.
Synthetic single-task fixtures, shared hardware and one trial per arm. No statistical or general superiority claim; canonical benchmark-store admission remains unverified.

Root-cause evidence ↗

Snapshot: 2026-10-16T13:22:22.547358+00:00
Source baseline: 75fb1c7a5473549983c16143e3cb15482c940d97