Bakeoff: More thinking isn’t (always) better

Turning coding agents up to high effort roughly doubled how much code they changed, and only 2 of 5 models solved more tasks with it. Fable 5.1’s default was its worst setting.

You’ve probably noticed the effort (aka “thinking” or “reasoning”) setting in your coding agent of choice. If you’ve thought about it at all, you set it by how hard the problem feels. A brief primer: thinking is the scratch work a model writes for itself before it answers, and the effort level sets roughly how much of it the model does. Here’s the effort level each agent ships with by default:

Table of the default effort level each agent ships with: medium for Haiku 5.5, Sonnet 5.5, and Opus 5.5 and high for Fable 5.1 in Claude Code, and low for GPT-6.1 Sol in Codex CLI.

Messy, right? In my case, I didn’t realize that GPT-6.1 Sol was set to “low” thinking for its first 3 weeks in Bakeoff, which hurt when doing comparisons to Opus. Ouch.

I ran 5 agents at up to three effort levels on real tasks (bug fixes and new features) from 5 codebases, graded by tests the agents never saw. Each agent ran each task 2 or 3 times at each level. The 5 codebases span several languages and shapes: 2 full-stack JavaScript web apps, 2 Python data pipelines, and a Rust tool that intercepts a client’s network traffic. Here are the results (full table in the appendix):

Line chart of task solve rate against mean cost per run, on a log scale, at each effort level for 5 models. Opus 5.5 rises from 79% at low to 88% at medium to 96% at high, and GPT-6.1 Sol from 75% to 79% to 88%. Sonnet 5.5 falls from 88% at low to 81% at medium and high, Fable 5.1 falls from 88% at low and medium to 75% at high, and Haiku 5.5 stays at 69% from medium to high. Every model costs more at each higher level.

Opus 5.5 and GPT-6.1 Sol got better every time I turned them up. Opus at high was the only setting to fix the hardest bug every time (3 of 3, against 1 of 3 at medium and max, and 0 for every other model), and Sol at high built a multi-user feature it missed every time at low. The other 3 just got more expensive. Sonnet 5.5 and Fable 5.1 solved fewer tasks at high (default for Fable!) then at low, and Haiku 5.5 stayed flat from medium to high.

At each model’s Claude Code default, Opus 5.5 beat Fable 5.1 on solves, cost, and time:

Opus 5.5’s medium default had a success rate of 88% for $1.07 a run with a median time of 3m10s
Fable 5.1’s high default had a success rate of 75% for $3.56 a run with a median time of 6m28s

Anthropic’s own write-up on effort found that effort doesn’t rescue a model that picks the wrong approach, which is what sank Fable on the hardest bug, where it missed the same cause on every run.

One side effect of increased coding agent usage is that the bottleneck in developing software is no longer coding speed, but instead human review to ensure software meets its requirements. Several engineering leaders I’ve talked to over the last few months have named human code review as the process that agents strain most. Code review was already no one’s favorite task before coding agents. High effort levels make the diffs (the code a reviewer has to read) 2.2x longer:

Bar chart of median lines changed per solved run at low, medium, and high effort for 5 models, split into code and the agent's own tests. Every model changes more at high: Opus 5.5 goes from 176 lines at low to 314 at high, Fable 5.1 from 188 to 322, Sonnet 5.5 from 41 to 258, and GPT-6.1 Sol from 100 to 231. Haiku 5.5, not run at low, goes from 195 at medium to 199 at high.

For runs with a working solution the mean diff goes from 126 lines at low to 281 at high. If you assume your engineers are reviewing 300 lines of code per hour and have 10 agent PRs a week to review, they’re going to spend 5 extra hours a week on just reviewing diffs from agents that spent their time writing longer solutions to problems that could’ve been solved with fewer.

I ran one last experiment on effort levels, giving Opus 5.5 three chances to run at “max” effort level on the hardest bug in the set. The result was that Opus did worse at max than it did at high:

Cartoon of Opus 5.5 as a robot at four effort levels on the hardest bug, 3 runs each. At low it thinks "ship it!" and fails all 3 with 6.4K output tokens. At medium it thinks "hmm, maybe two bugs?" and solves 1 with 19.7K. At high it thinks "aha, two bugs!" and solves all 3 with 35.1K. At max it sweats under a huge thought bubble ("what if... but then again... ok, one more refactor") and solves 1 with 231K.

My best guess on this failure is overthinking, which Claude Code’s own docs warn about at max: it rewrote far more code than at high (a median of 664 lines against 140).

What I’d set

If I were setting standards for a team today, here’s what I’d do:

Set effort level defaults per model.
Don’t default to max.
Fable 5.1: Set to low, if you use it at all.
Sonnet 5.5: Set to low. Gets an 88% success rate at $0.31 per run, under a quarter of the $1.39 Opus at high charges for more solves.
Opus 5.5 / GPT-6.1 Sol: Set to low for routine work and high for hard tasks.

Both tools let admins set an org-wide default to roll this out.

What I don’t know: the tasks I used aren’t yours, and your effort curves will be different. If you want to see yours, book 30 minutes with me at fairbake.dev and I’ll get you set up with a bake.

Thanks to Alex Monroe and Bryan Chen for contributing code and Jackie Hyman for editing!

Appendix

Solve rate, time, and cost by effort level

Model	Runs per level	Low	Medium	High
Haiku 5.5	21	n/a	69% · 2m49s · $0.047	69% · 3m45s · $0.068
GPT-6.1 Sol	21	75% · 2m38s · $0.18	79% · 3m58s · $0.22	88% · 5m44s · $0.33
Sonnet 5.5	21	88% · 1m33s · $0.31	81% · 1m29s · $0.35	81% · 2m50s · $0.70
Opus 5.5	21	79% · 1m49s · $0.58	88% · 3m10s · $1.07	96% · 4m18s · $1.39
Fable 5.1	16	88% · 4m07s · $2.33	88% · 5m45s · $3.11	75% · 6m28s · $3.56

Solve rate and cost is averaged over the 8 tasks, time is calculated as the median across tasks