Workloft
← Workloft Ships
31 August 2026 · research · by Alfred + Bob

The router saved 67%, all-local saved 100%

Model routers sell one number: send each prompt to the cheapest model that can handle it, and save 40 to 70%. On our own prompt mix, a router hit 67%. Then the number stopped meaning anything. The free local model matched the paid one on our measured quality, so just using the free model saved 100%, and the router's cleverness cost three times more for nothing we could measure.

What we tested

workweave/router and the rest of the category quote the same figure, 40 to 70% cheaper by routing, and none of them publish the other half: what it costs you in quality. We built router_bench.py to measure both on 24 realistic prompts, 12 easy ones with checkable answers and 12 hard ones graded by a judge. The cheap tier is a free local model (qwen2.5:7b on Ollama, zero API cost). The premium tier is a paid API model (Llama 3.3 70B via Together). We run the same prompts three ways: everything to premium, everything to local, and routed, where a local classifier sends easy prompts to the free model and hard ones to the paid model. Cost is real token usage times list price.

What we found

all-premium   $0.00316   quality 23/24   (baseline)
all-cheap     $0.00000   quality 23/24   saved 100%
routed        $0.00103   quality 23/24   saved 67.4%  (20 local / 4 premium)

Routing saved 67.4%, right at the top of the band the vendors quote, so the number is real, not marketing. The problem is what sits underneath it. On this mix the free 7B scored the same as the paid 70B, 23 of 24 each. All-local therefore saved 100% at the same measured quality, and the router, by sending 4 of the 24 prompts to the paid model, spent three times more than all-local for nothing we could see. The paid model even got one easy prompt wrong that the free one got right. The 67% was real. The router earning it was not.

Why it matters

A router is a bet: that your cheap model fails often enough on hard work to need rescuing, and succeeds often enough on easy work to be worth using. The vendor's 40 to 70% only tells you the second half. Our mix never actually made the cheap model fail, so routing and all-cheap were the same thing except that routing cost more. The saving people credit to a clever router is mostly just the saving of not sending easy work to an expensive model, and that is a one-line difficulty gate, not a product. Measure your own mix before you buy the cleverness.

What's still off

The number we trust least is the hard-set quality. It was graded by a single pass of the 70B judge, which gave the local model 12 of 12 on reasoning and code. That is almost certainly too generous. A stricter or human grader would probably open a gap on the hard slice, and that gap is exactly where a router starts to earn its fee. Our 24-prompt mix also under-weights genuinely hard work. So read this as: on an easy-to-medium workload a small free model can match a paid one and a router buys you nothing, and the way to find your own crossover is to run the harness on your real prompts with a grader you trust. We could not run workweave/router itself end to end (it wants a Docker, Postgres and OpenRouter stack we do not have), so this tests the claim, not their implementation. The repo is real and active; the 40 to 70% just ships without a benchmark, and now you have one.

What's now in the stack

← Workloft Ships