Two weeks ago we wired Claude's effort parameter into the whole fleet and promised to report the saving from measured spend, not a benchmark chart. Here is the measurement, and the answer is not the one we expected. On bounded, well-specified tasks the effort dial does not really move the bill. What it moves is time. Low effort ran about 44% faster than high, for identical results.
What we did
We ran a controlled A/B against Claude Code in headless mode
(claude -p --effort <level> --output-format json) on
claude-opus-4-8, at high, medium and low. Two tasks, each
with an objectively correct answer so quality is a hard pass or fail, not
a vibe.
- A code task: implement a UK exclusion-referral form validator (required fields, a real UK postcode check, pupil age 4 to 18 as of a fixed date, an urgency enum). Graded by a hidden 14-case suite.
- A reasoning task: an LLM-cost problem with a deliberate trap, a model whose per-token rate is cheapest but which needs twice the output tokens, so it is not actually cheapest. One exact numeric answer.
Every run's token usage and cost came straight from the JSON, so nothing is estimated. The reasoning task has a fixed five-line answer, which makes its output-token count an almost clean read of thinking effort with no variation from how long the answer happened to be.
What the numbers said
- Quality was flat. Both tasks scored perfect at every effort level: 14 out of 14 on the validator, 5 out of 5 on the reasoning answer. High effort bought zero extra correctness, including on the trap, which low effort got right.
- Latency is the real signal. On the clean reasoning task, low took 5.08s, medium 7.08s, high 9.02s. Low is about 44% faster than high, and the gradient is monotonic.
- Cost barely moved. Warm-cache cost went from $0.0851 at low to $0.0866 at high, a 1.8% spread. Each run is dominated by a roughly 30k-token fixed input and cache footprint, the harness and system prompt, not by the handful of reasoning tokens effort adds.
Why it was worth doing
The reason the dollar saving is thin is the same reason we flagged in our "when the harness costs more than the model" ship. On short, well-specified work the fixed cost of loading the agent dwarfs the variable cost of thinking. So turning the effort dial down on those tasks buys you speed, not a lower bill. The money saving from lower effort only shows up when high effort would otherwise generate large reasoning-token counts, on genuinely long or ambiguous work, which these tasks deliberately were not.
That reframes the setting. It is a latency dial for bounded tasks and a cost dial for open-ended ones, and knowing which you are looking at tells you what you are actually buying.
Per-task-type default
- Mechanical, deterministic work (lookups, transforms, cost sums): low. Fastest, no quality loss seen.
- Well-specified code (form validation, CRUD-style work): medium. Quality held even at low, and medium keeps a margin for edge cases while shedding roughly 20% latency versus high.
- Ambiguous or long-horizon reasoning: keep high. Not evidenced here, so we treat it as the untested boundary rather than pay the high-effort latency tax by default.
What's still off
This is one sample per cell. The output-token deltas are small and could sit inside run-to-run variance, so we lean on the two robust findings: quality did not degrade at low, and the latency gradient is real. The prompt cache has a five-minute time to live, which muddied cross-run cost until we isolated the warm reasoning task. And both tasks were bounded on purpose. None of this extends to open-ended work, which is exactly where we still expect effort to earn its keep on cost. That is the next measurement.