§1The pattern everyone is copying this month
Put a small, cheap model in front of your expensive one and let it throw out the obvious rubbish before you pay for the good model. That is the pattern of the month, and it is a good idea. Most work does not need a frontier model, so if a tiny local model can bin the clear failures for nothing, you only pay for the hard cases. On paper it is free money.
You can see it converging from three directions at once. Kev is an Apache-2.0 model family, sizes from about half a billion to nine billion parameters, sold as drop-in decision classifiers you can run yourself. Bending Spoons say they route ninety-nine per cent of their traffic through open-weight models and keep a frontier model for the top one per cent. Alibaba open-sourced a code reviewer that mixes fixed rules with a model and claims the same job at a ninth of the tokens. Different products, same shape: a fast cheap tier that answers most of the time, a slow expensive tier held back for what actually needs it.
We wanted to know what the cheap tier costs you when it is wrong. So we built one and measured it. That is the part nobody posts.
§2We built the cheap gate, then read its logits
Ours sits in front of Vera, our paid quality screen. Instead of asking the small model to write out a verdict, we frame the decision as a one-word answer, KILL or PASS, and read the probability straight off the option tokens of a frozen gemma3:4b in a single forward pass. No generation, no second call, effectively free. It only ever acts on a confident KILL, it never raises, and it can never add a wrong PASS. Code is open under MIT.
This is the elegant version of the pattern. One model, one pass, a number you can put a threshold on. It is exactly what the Kev-style classifiers are packaging, and it runs. In the narrow sense of "does it return an answer", it works.
§3The number that matters, 100% recall and 50% false-kill
Then we scored it against a small human-labelled golden set. Here is the honest headline, and it is more useful than "it works". On eight rows it caught every bad output. Perfect recall. It also killed half of the good ones: a perfectly valid summary and a correctly-empty result, both binned at full confidence, probability at or above 0.95.
Eight rows is a small set and a larger one would move the exact percentages. The shape of the error is the point, not the decimal. A gate that bins half of your good work is not a cost saving. It is a quiet way to throw away output you already paid to produce, and to do it without leaving a mark.
§4Confidence is not a safety signal
The tempting fix is a threshold. Only act on kills above some confidence, and let the uncertain ones through to the expensive model. It does not work here, and this is the part worth carrying to your own gate. The model's confidence did not separate its right kills from its wrong ones. It binned the good summary at the same 0.95 it used for the genuine failures. There is no line you can draw on that number that keeps the true kills and drops the false ones, because the number does not know the difference.
This is the trap in the whole pattern. A small model reading its own logits is fast, cheap and confident. Confident is the problem. It is confidently wrong on exactly the nuanced cases where you wanted a second opinion, and it reports that wrong answer with the same certainty as an easy one. The confidence score looks like a safety dial you can turn. It is not one. It is the thing that hides the mistake.
§5Wire the cheap gate to advise, not to act
None of this kills the pattern. The economics are real and the direction is right: cheap tier in front, expensive tier for the hard cases. The mistake is letting the cheap tier act.
So we shipped ours switched off. Not deleted, off. The gate runs, logs what it would have killed, and never touches the output. That does two useful things at once. It cannot bin good work, because it never acts. And every real decision the expensive model makes becomes a labelled row, which is the calibration set we actually needed instead of eight hand-built ones.
The rule we would give anyone building the same thing is short. A cheap gate is an adviser, not a judge, until it has earned the promotion on hundreds of real rows. Run it in shadow, compare it against the expensive model it is trying to replace, and look hard at the cases where it disagreed with confidence. Only move it into the kill path when the false-kill rate on real traffic is low enough that you would accept it deleting your own work. And never treat its confidence as the thing that keeps you safe.
The cheap gate is worth building. It is just not worth trusting yet.