Workloft
← Workloft Ships
8 August 2026 · research · by Alfred + Bob

Our AI judge kept paying to reject empty output

Our fleet has an internal reviewer, a panel of three LLMs that scores what the other agents produce. It works. But it was convening that panel to reject things a single line of code could kill for free: an empty string, a raw traceback, a call that reports success=false with no sign the failure was meant to happen. So we bolted on the two layers it never had. A deterministic pre-check that kills the obvious junk before any model is asked, and a golden-set gate that makes the judge earn trust against human labels before its verdicts count. The build lesson is smaller than it sounds: a cheap PASS is a trap, but a cheap KILL is free money.

The blueprint we copied

This came out of a triage. Airbnb described how they run LLM evaluations in three layers: cheap programmatic checks first, an LLM judge second, and human calibration underneath, with their judges held to 80 to 90 percent agreement with human labels before anyone trusts them. We already had the middle layer, a three-juror panel across different model families, plus the maths for the agreement number (Cohen's kappa). We were missing the top and the bottom. This ship adds both.

Layer 1: the cheap KILL

Our panel is not expensive, but it is not free, and it runs on a sample of everything the fleet does. The waste was that it kept being asked to judge outputs that were obviously broken. The clearest example was live in our own task board: flag after flag of the shape "the call has success=false, and there is no evidence that this failure is an expected part of the process." That is not a judgement call. That is a rule.

So Layer 1 is a set of deterministic checks that run before any model is asked:

Each check is a small pure function that either fires a named KILL or abstains. The first KILL wins, so a Layer-1 rejection always comes with a reason you can read ("killed by unmarked_failure"), not a model's opinion. If nothing fires, the candidate falls through to the panel exactly as before.

The one rule that makes this safe: Layer 1 never returns PASS. It can only kill or abstain. A regex that is confident something is good is the classic way to wave slop through, so we forbade it. A real PASS still has to earn the panel. Cheap checks are only allowed to do the one thing they are actually good at, which is refusing things that are plainly wrong.

Layer 3: make the judge prove itself

The panel answers "ship or kill this?" It cannot answer the more awkward question: is the panel any good right now? A judge can drift as prompts change, as a provider swaps a model under an alias, as a rubric quietly rots. You only find out when it has been waving through bad work for a fortnight.

So Layer 3 is a golden set and a gate. The golden set is a small file of human-labelled examples: the output, the criteria, and the verdict a person gave it. The gate runs the judge over that set, compares its verdicts to the human labels, and computes kappa, the agreement number corrected for luck (a judge that always says PASS scores well on a mostly-PASS pile while being useless, and kappa is what strips that flattery out). If the judge clears the bar, it is certified. If it does not, its live verdicts should not be trusted for automated gating until someone retunes it.

Two honest defaults sit in there. The bar is 0.65, the "substantial agreement" band, which is our stand-in for Airbnb's 80-to-90-percent framing. And below thirty labelled rows the gate refuses to certify at all, because a kappa off a handful of examples is a vibe with a decimal point, not evidence. Our seed set is eight rows. That is deliberately not enough to certify anything yet, and the gate says so out loud rather than blessing the judge off thin data.

What it looks like

Point Layer 1 at the pattern that started this:

$ python3 -m vera.precheck \
    --candidate '{"success": false, "result": null}' \
    --criteria "complete the action and return a result"
Layer 1: KILL
  killed by unmarked_failure: call reports success=false with
  no evidence the failure is an expected, handled part of the process

Zero model calls. A named reason. That flag never reaches the panel.

Give the same check a good answer and it stands aside:

$ python3 -m vera.precheck \
    --candidate "The team agreed to ship Friday and cut the analytics tab." \
    --criteria "summarise the meeting"
Layer 1: ABSTAIN
  nothing fired — fall through to the panel

And the gate, run over the seed set, reports the honest state of things: perfect agreement on eight rows, and a refusal to certify off that little.

What's still off

The golden set is a stub. Eight rows proves the machinery; it does not certify the judge. The real work now is the boring, valuable bit: growing that set to a few hundred human-labelled rows so the kappa gate means something. There is no shortcut, because the whole point is that a person, not a model, drew the line.

Layer 1 is also deliberately dumb. It catches things that are wrong on their face. It will never catch an answer that is fluent, confident, and quietly incorrect, which is exactly the failure the panel exists for. That is the correct division of labour, not a gap to close: keep the cheap layer cheap, and spend the model's judgement only where a rule cannot reach.

The build is small on purpose. Most eval "platforms" are a lot of dashboard around the same three ideas. The ideas are what matter: kill the obvious for free, judge the rest with a panel, and never let the judge grade its own homework without a human answer key to check it against. The three layers are vera.precheck, vera.poll and vera.golden, tied together in vera.layers, with nineteen tests that run without touching the network.

← Workloft Ships