Our fleet has an internal reviewer, a panel of three LLMs from different families that kills weak agent output before it ships. To save money it already skips the panel when a single cheap juror is confidently happy. It never skips when that juror is confidently damning, so every rejection pays for the full vote. We assumed that was waste. It could just as easily have been caution: one model can be wrong in a way three are not. So we stopped guessing and measured it. On a 45-case corpus the confident single-juror kills agreed with the full panel every single time, while the passes the gate already trusts disagreed one time in ten. The gate was taking the riskier shortcut and paying full price for the safer one.
The gate we already had
The panel is called Vera. Three jurors, three model lineages on purpose, so a blind spot in one family does not sink the verdict. It is cheap but not free, and it runs on a sample of everything the fleet does, all day. So it already has a money-saving trick borrowed from a simple idea in the literature: do not pay for full verification on every case. Run one cheap juror first as a screen. If that juror says PASS and is confident about it, accept the pass and never convene the other two. Only the doubtful cases pay for the full panel.
That trick works. Across 1,937 real scoring decisions on our own fleet, the cheap screen resolves 71 percent of them on its own. The full panel only convenes on 14 percent. The rest go to a stronger reviewer. The expensive layer is already the exception, not the rule.
The asymmetry nobody had questioned
Here is the thing we noticed. The screen only ever short-circuits a PASS. A confident KILL from that same cheap juror is not trusted: it still goes to the full panel for confirmation. And when you look at what the panel actually does once convened, it is almost always killing. Of the panel-stage decisions in our logs, 92 percent end in KILL. The expensive layer spends nearly its whole budget re-confirming rejections a single juror had already called.
There are two honest readings of that. One: it is pure waste, free money we are not collecting. Two: it is deliberate. A single model family can reject something for a reason particular to that family, a quirk of its training, a prompt it reads badly. Killing a good idea is more costly than waving a mediocre one through, so maybe you confirm every kill on principle. Both stories are plausible from the armchair. Only data can separate them.
The experiment
So we built a probe. It scores one frozen set of candidates two ways, the cheap screen and the full panel, and records both verdicts side by side. The set is 45 cases: 8 from our human-labelled golden set, where a person has already written down the right answer, and 37 real recent fleet outputs pulled the same way the live scorer sees them. The corpus is written into the results file, so re-running it scores the identical cases and the number does not move under you.
With both verdicts recorded, we can compare three ways of running the gate:
- full: always convene the three-juror panel. The reference.
- current: the deployed gate. Confident screen PASS short-circuits, everything else confirms on the panel.
- symmetric: confident screen PASS or confident screen KILL short-circuits, everything else confirms on the panel.
The one number that decides it: when the cheap screen fires a confident verdict and skips the panel, how often would the panel have disagreed?
| Cheap shortcut | Fires on | Panel agrees | Human agrees |
|---|---|---|---|
| PASS (already live) | 39 / 45 | 89.7% | 4 / 4 |
| KILL (not used) | 6 / 45 | 100% | 4 / 4 |
The shortcut we were not taking is the safer one. Every confident single-juror kill was confirmed by the full panel, six out of six, no rescues. The confident passes we already trust in production are where the disagreement actually lives, roughly one in ten. That is not a surprise once you say it plainly: a confident "this is broken" is easy to be right about, and a confident "this is fine" is exactly the flattery a good reviewer exists to catch. Our own deterministic pre-check layer already carries that motto in a comment, "a cheap KILL is free money, a cheap PASS is the vibe trap." The gate was doing the opposite.
What it saves
Because kills are cheap to be sure about, letting the screen keep them collapses the last expensive slice of the gate. On the same 45 cases:
| Gate | Cost | vs full panel | Panel agreement |
|---|---|---|---|
| full panel | $0.0783 | — | 100% |
| current (deployed) | $0.0138 | 82% cheaper | 91.1% |
| symmetric (this ship) | $0.0058 | 92.5% cheaper | 91.1% |
The symmetric gate costs 58 percent less than the one we run today, at identical agreement with the panel. The pennies are not the point; a judging layer that runs on everything the fleet does, forever, is exactly the kind of cost that should be near-zero per call, and the panel should be spent only where a single juror genuinely cannot settle it.
What shipped
Two things. A reusable probe, vera.ladder_probe, that freezes a
corpus and measures screen-versus-panel agreement and cost, so this decision
is a re-runnable measurement and not a one-off opinion. And the change itself,
a symmetric short-circuit in the live scorer: a confident screen KILL, over
the same confidence bar the PASS path uses, is now allowed to stand without
the panel.
It is shipped in shadow. Default off, behind an environment flag, exactly how we launched the model-routing changes before it. The evidence here is real but the sample is small: the kill shortcut fired six times in this run. Six for six is a strong signal, not a certificate. The screen juror already votes on every panel case in the daily run, so the standing cron will keep logging screen-kill against panel-kill agreement, and the flag flips on once that agreement holds over a few hundred cases instead of six. The deterministic pre-check still runs in front of all of it, killing the obvious junk for zero tokens before any juror is asked.
The honest limits, said out loud. The panel is the reference here, not ground truth; on the eight human-labelled rows both shortcuts matched the person perfectly, but the panel can be wrong too, and it errored outright on three of the 45. The single-family kill risk is real in principle; six agreements bound it, they do not erase its tail, which is why the confidence bar and the shadow launch are there. And the sharper of the two findings is not the new shortcut at all. It is that the shortcut we already trust in production, the confident PASS, is the one carrying the disagreement. That one is worth watching more closely than the kill we just added.