Bakeoff: Same fixes, 1.8x the bill, depending on the harness

Running 3 Claude models in Claude Code instead of pi cost 1.8x as much on 10 real tasks. The harness changed the bill, not the fixes.

A harness is the program around the model: its system prompt, its tools, and the loop that runs them. My scorecard compared agents as they ship, model and harness together. This time I held the models fixed: Opus 5.5, Sonnet 5.5, and Haiku 5.5, each under Claude Code, pi (a minimal harness with 4 tools), and opencode, twice each on the 10 scorecard tasks that still separate agents. That’s 180 runs, graded by tests the agents never see.

Two bar charts side by side for three harnesses running the same three Claude models on 10 tasks, 60 runs a harness. Tasks solved: 50 of 60 in Claude Code, 48 in pi, and 51 in opencode. What the 60 runs cost at API list prices: $19.88 in Claude Code, $10.99 in pi, and $13.59 in opencode.

Solves didn’t move. Claude Code fixed 50 of 60, pi 48, and opencode 51, and re-running Claude Code by itself 2 days apart moved as many results as switching harness did. What moved solves was the model: Opus 5.5 fixed 57 of 60, Sonnet and Haiku 46 each.

The bill did move. The same 60 runs cost $19.88 in Claude Code, $10.99 in pi, and $13.59 in opencode (at API list prices), and the gap grew as the model got cheaper: 1.7x for Opus, 1.9x for Sonnet, and 2.9x for Haiku.

Three bar charts of mean cost per run under Claude Code, pi, and opencode, one chart per model, with the tasks solved of 20 under each bar. Opus 5.5: $0.74, $0.42, and $0.52, solving 19 of 20 under all three. Sonnet 5.5: $0.21, $0.11, and $0.14, solving 16, 14, and 16. Haiku 5.5: $0.047, $0.016, and $0.019, solving 15, 15, and 16. Claude Code cost 1.7x, 1.9x, and 2.9x what pi did.

The run records show where the money goes:

A bigger starting prompt. Claude Code’s first request is about 19,100 tokens of instructions and tool definitions before any work, against 2,600 for pi and 6,800 for opencode, and every later request reads it again.
More round trips. Opus took a median of 17 requests per run in Claude Code, against 12 in pi.
Pricier caching. Claude Code caches for an hour, billed at 2x the input rate, and pi and opencode for 5 minutes at 1.25x. Even at 5-minute rates, Claude Code’s Opus runs would cost 1.44x pi’s.
A sandbox. Claude Code ran in its sandbox and blocked 124 commands across 45 of its 60 runs, and each block costs the model a turn spent finding another way. pi and opencode ran with no sandbox at all, so some of that cost buys real safety.

This lines up with HarnessTax, from Berkeley’s Sky Lab and Arena in September: on SWE-bench Lite, Claude Code cost about 2x what pi did at nearly the same success rate. My tasks are different (real bug fixes and features from 6 codebases, held-out tests), but the gap held.

How much of this reaches your bill depends on your team’s contract with Anthropic or Bedrock. If you pay per token, the harness cost difference is something you can solve: switching from Claude Code to pi cut cost per run by 43% to 66% here. If you pay per seat with usage included, the cost difference won’t show on the invoice until people run past their limits.

If I were setting standards for a team today:

Pick the model first. It moved solves here and the harness didn’t.
If you pay per token, try a lighter harness on your own tasks before you try a cheaper model.

If you want to know what your harness costs you on your own code, book 30 minutes with me at fairbake.dev and I’ll get you set up with a bake.

Thanks to Alex Monroe and Bryan Chen for contributing code!