Every company buying coding tools and agentic workflows right now is deciding on vibes. I spent the last month building a private eval on code no model has seen. Here is what building it taught me, and what I found out.
Quick context, since last week’s post was the teaser: Bakeoff takes real bug-fix or feature commits from my own repos, hides the tests that prove the fix or new feature worked, hands a prompt and the broken tree to a coding agent, and grades whatever comes back against the hidden tests.
By the numbers:
- 145 recorded runs across 7 entrants (harness x model combinations), spanning 7 models and 3 harnesses (Claude Code, Codex CLI, and pi), at an approximate cost of $80 at list API rates.
- Bakeoff repo: 21 merged PRs, 92 commits and about 6k lines each of design docs / Go code
- 7 bug-fix or feature commits, each with a handcrafted prompt (a “challenge”), mined from 3 apps I wrote and run: two Express web apps and a Python agent that runs as a cron job. None of this code is public, so no model has seen the fixes.
I might have hand-written 10 lines of code in total. My month instead went to thinking through new features, iterating on designs my agent drafted, watching how the implementation behaved, and reviewing code before merge. This isn’t a new way of working for an experienced engineer. It’s roughly the same split I’ve had for the last five years, minus the part where I typed the code.
Building Bakeoff
What kept quality high
Not writing the code only works if something else catches the mistakes. Agents thrive with guard rails, ideally deterministic ones. Bakeoff ended up with 78 unit tests, gofmt running on every push, CI that strictly checks configuration YAML, and an AGENTS.md that standardized the rest: design docs before code, a formalized roadmap, branching and PR rules, and manual test steps written down so they can be re-run.
The same instinct applies to the agent eval itself. Every run keeps its full session transcript and output tree on disk, which has paid for itself many times over in debugging broken harnesses, and it’s where questions like “do agents that write their own tests do better?” get answered.
I picked tools that made those guard rails cheap. Go is a great language for agents - type safe, succinct, batteries included, and lots of examples in the training data. It also feels nice after years of dealing with legacy Java and C++ codebases.
Where the agent needed me
The above sounds nice, but none of my plans have ever survived first contact with a coding agent. Here’s some of the taste I added during development, mostly with Claude Code and Fable 5.1:
- Preventing the answer key from leaking - Claude added a revert.paths field to each challenge, listing which files in the fix commit to roll back before handing the tree to the agent under test. That fails dangerously: forget one path and the agent sees the fix. I flipped it to start from the pre-fix commit and list only the tests to keep.
- Rejecting a requirement for “guard” tests - The initial design required every challenge to name tests that catch an over-correction, like an agent that fails every request so the “reject bad requests” test goes green. I pushed back. It’s a nice way of categorizing failures, but it would have significantly slowed down the rate of adding challenges, which mattered more.
- Deleting a check that only caught one bug - A side project of mine had a test failing at HEAD because it compared today() to a fixed date. Claude’s design wanted to detect that specific mistake and reject such tests from challenges. That check added complexity without solving the general problem of time-dependent tests, so I removed it.
- Deleting code history embedded in comments - When I corrected something, Claude would leave both the original rationale and the correction in the comments. That’s duplicative and harder to read, and the commit history already records the mistake.
Software is still hard
The agent sandboxing problem was harder than I’d planned for. pi reads instruction files from parent directories, even when told to stay in its work directory. Claude Code’s default sandboxing broke uv, which needs a cache directory to work. Building a consistent set of adapters for each agent harness was a grind of small details in config files, environment variables, and flags. And one of those details really mattered (see Results).
Results
Once the harnesses stopped fighting me, the runs started giving actionable intelligence. 145 of them so far with 7 entrants running across 7 challenges. Because entrants are harness × model combinations, Fable vs Astra is also Claude Code vs Codex. GLM-5.2 was run via OpenRouter. Four things stand out.
The same bug fix costs 2 cents or $2, depending on who does it. The spread in cost between green solutions is over 80x: GPT-5.6 Luna in Codex and GLM-5.2 in pi at the bottom, Opus 5 and Fable 5.1 in Claude Code at the top. The newest frontier entrants use 10% fewer output tokens than their predecessors, but their per-token rate more than eats the difference.
This cost difference is a significant problem on the work everyone wants to automate: an attempt on every failing CI build, flaky test, unstaffed ticket, and failed request. A thousand of those a day is thousands you didn’t need to spend.
Databricks saw the same thing on their own multi-million-line codebase in July (link): an open-weight model matched their frontier model’s completion rate at two thirds the cost per task.
Speed varies by entrant moderately, and it sorts by harness, not model. Across an 80x spread in cost, the slowest entrant takes only 2x as long as the fastest: 1m33 for GPT-5.6 Sol to 3m12 for Opus 5. Every Codex entrant finished ahead of every Claude Code entrant. Within Claude Code, Fable 5.1 beats Opus 5 by about 20%, and the three Codex models are within 20% of each other.
The newest models write tests without being asked. GPT-6 Astra touched the test suite on 12 of 14 runs and Fable 5.1 on 6 of 14. Every other entrant did 3 times or fewer. It’s the first behavioral split in the data, and it didn’t change who solved what - yet. These challenges are small enough that a careful diff gets you there. The harder ones are next.
A sandbox setting prevented a harness from running tests. Codex’s default sandbox blocks the test server from starting by preventing it from binding a port, so on 6 of 7 challenges its models never saw any integration tests go green. They improvised with one-off scripts instead, and mostly got away with it. But the only two failures in the data are both Codex, and both were slips a single test run would have caught. It took two full sets of runs and a read through the transcripts to notice. A healthy success rate hid an agent working with one hand tied behind its back.
Three rules for your team
Confirm the agent can run the test suite green before comparing anything. If it starts writing one-off scripts or installing dependencies instead of running your tests, something is in the way, and every result after that is measuring the wrong thing.
Run the cheap model first, escalate on a red test. On these tasks GPT-5.6 Luna solved 14 of 15 them for about 1/80th of the price.
Keep every session transcript. The port-binding failure above was invisible in the scores and obvious in the logs. The questions you’ll want to ask next month are only answerable if the sessions are on disk.
What’s next for Bakeoff
The current challenges are as unrepresentative as they’ll ever be and the numbers are only a few runs deep. The roadmap is about fixing both, starting with the items that matter most if you’re picking tools for a team:
Broader and harder tasks. Fixes or new features that span domains, services, and/or repos, where the symptom or visible output is far from the relevant code and a plausible wrong implementation is likely. This is where correctness rankings separate frontier models and where you start to be able to trust your most critical business systems to agents. Maybe even without code review!
What are you actually paying for? Three experiments on the cost axis. A survey of cheaper open-weight models, to see what you’ve been missing if you’ve only run Codex or Claude. The same model in different harnesses, to find out whether pi’s lean turn loop is a token saving or a quality loss. And thinking levels, which most teams leave at default without knowing what it costs or buys them.
Does your AGENTS.md matter? Same model, same tasks, different instruction files: coding style rules, SDLC rules, none at all. Teams argue about these for hours. The eval can settle it in one set of challenge evaluations.
Bring me your commits
If your team is picking agentic tooling on vibes, bring three recent commits to a 30-minute call: bug fixes or features, each with the test that proved it. I’ll walk through what a challenge evaluation on each would measure, what it would take to run inside your environment, and what the results above suggest before you run anything. Message me on LinkedIn or send me an e-mail.
Thanks to everyone who reached out since the teaser, and especially those who’ve met me directly. Those chats are shaping which way Bakeoff evolves.
Appendix
List of entrants
| Entrant | Harness | Model | Runs | Solved |
|---|---|---|---|---|
| Luna | Codex CLI | GPT-5.6 Luna | 15 | 14 |
| GLM-5.2 | pi | GLM-5.2 (via OpenRouter) | 17 | 17 |
| Sol | Codex CLI | GPT-5.6 Sol | 10 | 9 |
| Astra | Codex CLI | GPT-6 Astra | 14 | 14 |
| Sonnet 5 | Claude Code | Claude Sonnet 5 | 16 | 16 |
| Opus 5 | Claude Code | Claude Opus 5 | 10 | 10 |
| Fable 5.1 | Claude Code | Claude Fable 5.1 | 14 | 14 |
Results caveats
Costs are measured as real OpenRouter spend for GLM-5.2 and list API prices from Claude Code and Codex. Timing measures wall-clock time, including the vendor’s serving latency and the harness’s turn loop as well as the model. Only 96 runs are used in the analyses above since the first 49 ran under an earlier format. The number of runs per entrant, 10 to 17, differs because Fable 5.1 and GPT-6 Astra were added after they were released, and I stopped re-running Opus 5 and Sol on older challenges after their successors were added. Every entrant has at least one run on every challenge.