Why does your AI coding CLI pump 100,000 tokens into the model just to edit a 50-line file? We inspected the raw payloads of Claude Code, Codex, Devin Fusion, and Verdict to find out.
Monolithic CLIs charge you for tens of thousands of tokens of unused tool schemas on every single turn. Slide to see how much your engineering organization saves with Verdict.
By switching from bloated CLI tool schemas to Verdict's lean agent harness, your team eliminates 105.3M in wasted framework tokens every single month.
3 trials × 2 tasks (Ember & Oak landing page + interactive task manager) evaluated across 78 automated Playwright test assertions outside the repository to eliminate prompt contamination.
| Arm & Architecture | Pass Rate | Wall Time (mean/med) | Tokens In / Out (mean) | Cost / Run |
|---|---|---|---|---|
Verdict CloudMost Efficient Sonnet 5 (Direct BYOK) | 98.7%(77/78) | 40s / 39s | 4,849 in4,874 outLean harness (20.7x fewer in) | ~$0.0584 |
Verdict Local100% Free / Private Qwen3-Coder-30B | 92.3%(72/78) | 51s / 45s | 11,682 in2,862 out | $0.00 ~2.9 Wh / run |
Claude Code Sonnet 5 | 100.0%(78/78) | 32s / 32s | 100,569 in3,769 out52K cache write + 48K cache read | $0.1624 |
Codex CLI OpenAI GPT-5 / Sonnet-5 Proxy | 100.0%(78/78) | 128s / 126s | 67,996 in3,380 out | ~$0.1698 |
Devin Fusion Fable 5.1 Lead + SWE-2 | 97.4%(76/78) | 110s / 97s | 86,018 in3,678 out | ~$1.0441 |
We pulled the raw out.json telemetry files. This is not a token-counting discrepancy — it is a structural architectural divide.
{
"num_turns": 2,
"usage": {
"input_tokens": 4,
"cache_creation_input_tokens": 52326,
"cache_read_input_tokens": 48344,
"output_tokens": 3983
},
"total_tokens_billed": 104657,
"total_cost_usd": 0.2588108
}52,326 tokens written to cache on Turn 1, then 48,344 tokens read on Turn 2. The payload is dominated by Claude Code's built-in system prompt and full JSON schemas for 15+ built-in tools (Read, Write, Edit, Bash, Glob, Grep, Task, TodoWrite, etc.), resent regardless of task scope.
{
"num_turns": 2,
"usage": {
"input_tokens": 4849,
"output_tokens": 4874,
"cache_creation_input_tokens": 0,
"cache_read_input_tokens": 0
},
"total_tokens_billed": 9723,
"total_cost_usd": 0.058428
}Full multi-turn session summed via _aggregate_terminal_number(). Verdict uses a lean, right-sized agent harness. No monolithic schema dumping, no 50K-token cache tax. 20.7x fewer input tokens, delivering 98.7% accuracy for a fraction of the cost.
Real benchmark integrity means showing exact failure modes instead of cherry-picking 100% runs. Here is why Devin Fusion and Verdict Local missed checks in the rerun.
<nav id="nav">
<!-- Extra link added inside nav element -->
<a class="brand" href="#hero">Ember <span>&</span> Oak</a>
<ul>
<li><a href="#hero">Home</a></li>
<li><a href="#menu">Menu</a></li>
<li><a href="#about">About</a></li>
<li><a href="#contact">Contact</a></li>
</ul>
</nav>Devin Fusion included the brand logo link inside the <nav> element alongside the 4 navigation links, resulting in 5 links. The prompt asked for 4 nav links, and the automated grader strictly counts links inside #nav.
tasks.forEach((task, index) => {
if (currentFilter === 'active' && task.completed) return;
if (currentFilter === 'completed' && !task.completed) return; // <-- skipped from DOM
const taskElement = document.createElement('li');
// ...
taskList.appendChild(taskElement);
});Verdict Local implemented 'filter' by re-rendering only matching tasks rather than toggling CSS visibility. While semantically intuitive, the grader asserts that non-matching items remain in the DOM with hidden styles. Smart Harness retry policies detect and patch this class of DOM divergence.
How Verdict delivers frontier reasoning quality with 20x less token bloat.
Instead of dumping 15 large JSON schemas for every possible file, bash, and task operation into the prompt, Verdict scopes tool definitions dynamically. Models get what they need to execute the current step, freeing context for your actual codebase.
Verdict Cloud connects directly to Anthropic, OpenAI, or Z.AI with your own API key. There is zero framework markup, no hidden proxy rerouting, and no inflated prompt taxes. You pay exactly what Anthropic charges for Sonnet 5 — which is ~\$0.058 per run, not \$0.16 or \$1.04.
Why send your code to third-party cloud servers at all? Verdict Local connects out-of-the-box to Apple Silicon (MLX), LM Studio, and local GPUs. Run Qwen3-Coder-30B with 92.3% pass rates at zero marginal cost and total privacy.
Claude Code is a monolithic CLI framework that bundles large, comprehensive system prompts and exhaustive JSON schemas for all its built-in tools (Read, Write, Edit, Bash, Glob, Grep, WebFetch, TodoWrite, etc.). In turn 1, it writes over 52,000 tokens to prompt cache, and in turn 2, it reads back 48,000 tokens. This overhead is fixed per session and fires even when editing a 10-line file.
No. We verified the extraction code directly in `tools/verdict-code/verdict_code/main.py:5550` (`_aggregate_terminal_number`). Verdict explicitly sums the input and output tokens across every turn in the entire session. Verdict's low numbers are real because its harness is built lean: minimal system prompt overhead, targeted tool schemas, and modular execution.
Cached read tokens are discounted (typically 1/10th base price), but they are not free. Furthermore, cache-creation tokens (the initial 52K) are billed at full price on every new session. Most importantly, pumping 100K tokens into context increases time-to-first-token, increases latency, burns rate limits, and crowds out project context.
Verdict uses an intelligent microservices-based agent harness that injects only the schema and context necessary for the current task phase. Using direct BYOK (Bring Your Own Key) to Anthropic Sonnet 5, you get frontier reasoning quality without paying for framework bloat.
Yes. Verdict Local connects directly to local inference backends (LM Studio, MLX, Ollama, vLLM). In our head-to-head rerun, Qwen3-Coder-30B achieved 92.3% test pass rate at 51 seconds wall time, with zero API token costs and just ~2.9 Wh of energy consumption on Apple Silicon.
Run Verdict on your local machine with your own models, or connect direct BYOK for frontier intelligence without the framework bloat.