On two real jobs from our agent fleet, Claude Haiku 5.5 beat the cheap models we had been using, and did not cost more. It scored news items closer to Claude Opus 5 than Gemini 2.5 Flash did, and it edited a digest with far fewer made-up facts than DeepSeek V4 Pro. It now runs both jobs, with the old models kept as fallbacks.
What we tested
Haiku 5.5 came out on 7 October at $0.10 per million input tokens and $0.50 per million output, for prompts under 100,000 tokens. Price cuts are easy to announce, so we ran it on work we already do, with the production prompts and real inputs from this week.
- Scoring. Our Agentic Oddities digest scores every candidate story 0 to 10 before an editor picks the best. We re-scored all 141 items from Wednesday's run. Opus 5 scored them too, as the reference.
- Editing. The editor reads a shortlist and picks a headline story, three mentions and a "news angle" for each, without inventing facts. We used the last three real shortlists. Opus 5 judged the outputs blind.
Each job ran on the incumbent and on Haiku 5.5 at low and at high effort.
What we found
- Scoring: Gemini 2.5 Flash, run as we run it in production, landed 1.34 points from Opus on average. Haiku 5.5 landed 0.49 points away at both effort levels. On the shortlist (stories Opus scored 6 or more) Haiku at high effort scored F1 0.67, Gemini 0.57 and Haiku at low effort 0.50.
- Cost of scoring: the batch cost $0.038 on Gemini at Google's list price, $0.032 on Haiku high and $0.021 on Haiku low.
- Editing: the judge gave DeepSeek V4 Pro 6.3 out of 10, Haiku low 7.7 and Haiku high 8.7. The gap was grounding: 5.7 against 9.7. In the judge's words, DeepSeek "fabricates mechanisms wholesale". DeepSeek still made slightly better picks.
- Cost of editing: three runs cost $0.019 on DeepSeek and $0.012 on Haiku high.
Low effort was cheapest but returned 4 broken JSON replies out of 141 and missed more of the stories that mattered. High effort fixed both, so that is what we run.
What we changed
Our model router now sends classification work and a new digest-editing route to Haiku 5.5 at high effort, with Gemini and DeepSeek as fallbacks. Haiku 5.5 rejects a temperature setting, so the router now drops it. While pricing the test we also found the router had Gemini Flash at a quarter of Google's real input price and an eighth of its output price, so our own spend reports had been reading low. That is fixed too. At current volume the switch costs us roughly $2 to $3 a month.
What's still off
The reference and the judge are both Claude, so a family bias is possible. The next round adds a judge from another lab. The samples are small: 7 shortlist stories and 3 editing runs. Haiku 5.5's tokeniser used about 50% more input tokens than Gemini on the same prompts, which the per-token price absorbs here but not everywhere. Above 100,000 tokens a prompt costs five times as much. And we tested two jobs, not the whole fleet.
Take it
The test script, the result file and how to point it at your own prompts are on GitHub. Run your own work through it before you believe anyone's leaderboard, ours included.