Is Gemini 3.8 Flash a cheap model for coding?

It is cheap per input token, but its output tokens are priced high (reported at $3.75 per million). On routine coding tasks the model writes short answers where output dominates the bill, so on our measured mix it cost 2.4 times more than a paid open 70B (Llama 3.3 70B on Together) at the same quality. Cheap headline input price is not the same as cheap for your workload.

Does a cheap model match a bigger one on coding quality?

On easy-to-medium coding tasks with deterministic unit tests, yes for the cloud tiers. All three cloud tiers passed 8 of 8 (Gemini 3.8 Flash, Llama 3.3 70B, and Claude Opus 4.8), across a 23x cost spread, so cost separated them, not quality. A free local model (qwen2.5:7b) was too slow on a CPU box to be viable, timing out on 7 of 8 tasks. Genuinely hard tasks would likely open a quality gap, which this bench does not test.

The cheap frontier model that cost the most

Q: How do you benchmark model coding quality without an LLM judge?

Ask each model for a function with a fixed signature, extract the code from its response, and run it against hidden unit tests in a subprocess. Pass means every assertion passed. The quality number is a pass rate with no grader model in the loop, so it carries no judge bias.

Google's Gemini 3.8 Flash is sold as cheap frontier coding. We put it on a deterministic coding bench against a paid open 70B and a free local model. All three got the same answers right. Flash cost 2.4 times more than the 70B and ran three times slower, because its output tokens are priced high. On routine coding work, the cheap frontier model was the expensive one.

What we tested

Gemini 3.8 Flash landed this week with the usual framing: near-frontier coding at a fraction of the cost. The question we cared about is narrow and practical, because it is the one that decides whether a model earns a slot in our fleet: is it good enough, and cheap enough, to be the tier that does the boring high-volume coding work (write the function, run the tests, fix the lint) while an expensive model handles judgement? So we built flash_bench.py: 8 self-contained Python coding tasks, each asking for one function with a fixed signature (parse a duration string, merge intervals, Roman numerals, run-length encode, and so on). Quality is graded the honest way, with no judge model in the loop: we extract the code from each response and run it against hidden unit tests in a subprocess. Pass means every assertion passed. We ran it across three tiers you would realistically route cheap work to: Gemini 3.8 Flash, a strong paid open model (Llama 3.3 70B on Together), and a free local model (qwen2.5:7b on Ollama). Cost is real token usage times list price.

What we found

tier                 pass    cost $     total tokens   time
Llama 3.3 70B        8/8     0.001365   1551           13s
Gemini 3.8 Flash     8/8     0.003276   1324           39s
Claude Opus 4.8      8/8     0.031905   1949           18s
qwen2.5:7b (local)   1/8     0.000000   free, but timed out on 7/8 on CPU

Quality did not separate the three cloud tiers: they all passed 8 of 8. The free local model was a different story, and worth being honest about, we shipped an early "it passes" read off a partial run and the full run corrected us: on a CPU box qwen2.5:7b was so slow it timed out on seven of the eight tasks (a five-minute per-call limit, three retries each), completing only one. Free in dollars, unusable in practice without a GPU. Between the cloud tiers the separation was pure cost, and it spanned a factor of 23 at identical measured quality. Gemini 3.8 Flash cost 2.4 times more than the paid 70B. Claude Opus 4.8, the frontier model, cost 23 times the 70B and roughly 10 times Flash, and got the same eight answers right. On this workload the entire cloud price ladder bought nothing you could measure.

For Flash specifically, the reason it lost to the 70B is the price sheet, not the model: it is cheap on input (reported at $0.75 per million tokens) but expensive on output ($3.75 per million), and on short coding answers the output side is most of the bill. A flat-priced commodity 70B undercut the cheap frontier model on the exact work the frontier model is sold for. The free local model, meanwhile, never got a fair fight on this hardware: it timed out on seven of eight tasks, so "free" came with "cannot actually finish the work" on a CPU box.

Why it matters

"Cheap model" is a headline about one number, usually the input price. Your bill is a different number: your token mix times the full price sheet. Coding subagents generate short, dense output (a function, a diff, a test), so they are output-weighted, and an output-weighted workload run on a model with a high output price is not cheap however cheap the input is. The wider point is the ladder: on routine coding a paid 70B, a cheap frontier model and a full frontier model all returned the same answers, so a 23x cost spread across the cloud tiers bought nothing measurable (the free local model could not keep up on CPU at all). Frontier models earn their price on hard problems, which is where you should spend it, not on the boring high-volume work a cheap tier does just as well. The lesson is the same one the router taught us last week: the marketing number is real and still tells you almost nothing about your bill. Price the model on your own workload's token shape, and match the tier to the difficulty of the task, before you believe the word cheap.

What's still off

The load-bearing caveat is task difficulty. These 8 tasks are easy-to-medium and deterministic, and every cloud tier passed, so this bench measures cost and latency on routine work, not hard reasoning. Genuinely hard problems are exactly where a frontier model would open a quality gap and start earning its output price, and we do not test that here, so do not read this as "Flash is worse." Read it as "on the boring high-volume coding a cheap tier actually does, Flash's price advantage disappears." Second, the Gemini price is as-reported from launch coverage, not a contract we have run a bill against; the harness prints real token counts so you can plug your own rate. Third, N is 8 tasks, single-pass. All four tiers, including Claude Opus 4.8, are measured on the same bench; the one comparison this does not make is a like-for-like on genuinely hard tasks, which is the next run and the place the price ladder should finally mean something.

What's now in the stack

flash_bench.py: deterministic, no-judge coding bench across a cheap-frontier (Gemini 3.8 Flash), a paid-open (Llama 3.3 70B), a full-frontier (Claude Opus 4.8) and a free-local (qwen2.5:7b) tier, with real cost per tier.
A retry-on-call-error guard, so a flaky API response is retried rather than misreported as the model getting a task wrong.
The full run log and result.json behind the numbers above.
The kit on GitHub. Point it at your own tasks and your own contract prices.