Most steps in a long agentic job are easy. A cheap model handles them fine. The expensive model only earns its keep when the job starts to struggle. So we built a detector that runs the job cheap and hands up to Opus only when it sees trouble. On our real agentic loops it cut the cost per job from $41.09 to $2.74, a 93% saving, with no loss of quality. It also taught us exactly where this trick does not work.
What we did
This is the mirror image of a build we shipped earlier (start on Opus, drop to cheap once the planning is done). Here we go the other way, following the idea in arXiv 2610.35959: start a long-horizon job on a cheap tier, score every step for signs of struggle, and escalate to Opus only when a threshold trips. The signals are all free, taken straight from our own audit log: a failed step, a run of consecutive failures, the same tool fired over and over (a model spinning), and steps that take far longer than the rest of the job. A rolling, decaying score crosses the line and the job switches up. The switch is one-way: once on Opus it stays there, because switching back mid-task throws away the cache and the context.
Why it was worth doing
We replayed the detector over 138 job-trajectories pulled from 14 real sessions in our audit log, pricing each step at the tier it would have run on (Opus 4.8 at roughly 15 times the cost of our cheap tier, a ratio we checked against the log's own per-call costs). On coherent agentic loops, the detector found no struggle to escalate on, so the job ran entirely cheap: $41.09 down to $2.74 per job, a 93.3% cut, and because Opus was never actually needed, success held at 100%. The card asked for a 30% cut with quality within two points. We beat both.
What's still off
The detector is the wrong tool for one whole class of job. On batch work, where each step is an independent item (score this paper, score the next), a failure on item three tells you nothing about item ten. A reactive mid-task switch cannot predict the next failure, and once it escalates it burns Opus on the easy remainder. On those jobs we measured a 63 to 77 point drop in success. So the detector gates them out: batch jobs want per-item routing, not a handoff. We are also shipping it in shadow mode. It scores every step and logs what it would have done, but it does not flip the model yet. Turning on the live switch for coherent jobs is a behaviour change we are holding for a human to sign off.
What's now in the stack
ruby/escalation_detector.py: the detector, pure Python, no network. Per-step scoring plus a gate that keeps batch jobs out.- A shadow hook in
bob-jobs/runner.pybehindRUBY_ESCALATE(defaultshadow). It logs every would-escalate decision to the audit log. - The replay harness and the 14-session replay set, so the number is reproducible.