§1What we run, and what it decides
Workloft has a gate called Vera. Ideas go in, and Vera decides which ones are worth a build slot. It is not advisory. A KILL means the thing does not get built, so a lazy judge costs us real work, in both directions: a soft judge burns a week on a bad idea, and a harsh one throws away a good one and nobody ever finds out.
Vera is a panel, not a single judge. Three jurors vote, deliberately drawn from three different model lineages so that a blind spot in one family does not quietly become the house view:
- Anthropic Haiku 4.5
- Google Gemini 2.5 Flash
- DeepSeek v4 Flash
Three PASS is a pass. Three KILL is a kill. A two-to-one split escalates to me. The whole panel costs about $0.002 a candidate, which is the point of the design: the research on panels of LLM evaluators says several small judges from different families beat one big expensive one, largely because their errors do not line up. That is a claim about correlation, and it is the reason the lineups are hardcoded.
We had been running it for weeks. What we had not done was read the LLM-as-a-judge literature against our own source file, line by line, asking which known failure mode we had actually left in. So we did that. Here is what it cost us.
§2The bug that mattered: we collected confidence and then ignored it
Every juror returns a verdict and a confidence between 0 and 1. We collected that number on every call, logged it, and then, at the decision, threw it away.
The split path is where a judge panel earns its keep, and it was the path we had been sloppiest about. When a two-to-one split could not escalate to a human, the old code resolved it by counting heads:
# before if winner == "TIE": result.verdict = "KILL" result.confidence = 0.4 else: result.verdict = winner result.confidence = min(win_conf, 0.6)
Read that again with the confidences in mind. Two jurors mumble a PASS at 0.55 apiece while the third screams KILL at 0.99, and the panel returns PASS, because two is more than one. We had the evidence that the majority was unsure and the minority was certain, sitting right there in the objects, and we discarded it.
The fix sums each side's confidence rather than counting its heads, and takes the margin seriously:
# after pass_score = sum(v.confidence for v in pass_votes) kill_score = sum(v.confidence for v in kill_votes) margin = abs(pass_score - kill_score) if margin < 0.25: # genuine toss-up result.verdict = "KILL" # bias toward killing result.confidence = 0.4 else: result.verdict = "PASS" if pass_score > kill_score else "KILL" result.confidence = min(0.6, 0.5 + margin / 2)
Now the confident minority can beat the hesitant majority, which is the entire reason we asked for confidence in the first place. A near dead-heat still resolves to KILL, because when the panel genuinely cannot separate an idea from a bad one, not building it is the cheaper error. And note what we did not touch: the unanimous paths, and the escalate-to-a-human path. Those were working. An audit that rewrites the parts that were fine is not an audit, it is a rewrite with a nicer name.
§3Two prompt fixes that cost nothing
Reason before verdict
Our jurors return strict JSON. The schema used to put verdict first and rationale second. That is backwards, and it is backwards for a mechanical reason, not an aesthetic one: a language model writes in order. If the verdict token comes out first, every word of the rationale that follows is generated conditioned on a decision already taken. That is not reasoning. It is a press release.
We swapped the two keys, so rationale is now the first field the model must emit, and told it plainly: reason first, then commit to a verdict, do not decide and then justify. The cost of this change was moving two lines in a docstring.
Judge the substance, not the surface
The well-replicated embarrassment of LLM judges is that they reward the wrong things: length, formatting, and a confident tone. A padded, well-organised, authoritative-sounding answer beats a short correct one, and the judge will happily explain why. Our jurors had no instruction guarding against this at all. They now open with one:
Judge the SUBSTANCE, not the surface. Do not reward length, formatting, or a confident tone: a short correct answer beats a long padded one, and "sounds authoritative" is not evidence. Penalise unnecessary verbosity.
Neither of these is clever. Both are the kind of thing you only find by reading your own prompt as though a stranger wrote it.
§4We were measuring the panel with the wrong number
The last fix is the one I would most want another builder to steal, because it is the one that tells you whether any of the rest worked.
We had been tracking how often Vera agreed with me. That number is close to worthless on its own. Agreement rate flatters a judge on imbalanced data: if 90% of candidates deserve a PASS, a judge that says PASS to literally everything scores 90% and has learnt nothing. It is not a measure of skill, it is a measure of the base rate.
So we now compute Cohen's kappa, which subtracts the agreement you would expect from chance and reports what is left. The action rule is written into the file so nobody has to relitigate it later:
| Kappa | What we do |
|---|---|
| ≥ 0.65 | Panel runs automated |
| 0.60 to 0.65 | Watch, keep labelling |
| < 0.60 | Halt automation, retune the rubric |
The calibration rule, from vera/calibrate_kappa.py. The script also reports n, because the number below is why that matters.
And here is the honest state of it: we have not yet earned a kappa worth quoting. Our labelled set is six rows, from my first review pass, and the panel agreed with me on all six. Six is not a sample, it is an anecdote. A perfect score on it means nothing, and the script exists precisely so that we cannot go on quoting a flattering agreement rate at ourselves while we build the labelled set that would actually settle it. We built the ruler before we liked what it said. That is the correct order.
§5Two things we did not change, and why
Audits have a failure mode of their own, which is that once you have the research in hand, everything looks like something to fix. Two candidates came up, and we left both alone on purpose.
Gemini is both our screen and one of our jurors. Before convening the panel we run a single cheap Gemini call as triage, to decide whether a candidate is even worth three votes. Then Gemini votes. That dual role looks exactly like the self-preference bias in the literature, where a judge favours text it generated itself, and the tidy move would have been to swap the screen to a fourth model and write a paragraph about how rigorous we are. We did not, because the premise does not fit: Gemini is not marking its own homework here. It never generated the candidate. It is re-reading a proposal it has already seen. The real risk is anchoring, which is a different mechanism and needs a different test, and we have not run that test. So it stays on the list as an open question rather than getting a fix aimed at the wrong bias.
We did not add more jurors. The obvious "more rigour" move is a bigger panel, and the panel-of-evaluators result is the sort of headline that invites it. But the replication evidence for these panel effects is thinner than the abstract suggests, the gains beyond a small panel are not clearly real, and every extra juror costs money on every call and risks re-introducing the correlated failure the design exists to avoid, if the new juror shares a lineage. Three families, stop. Adding jurors would have felt like progress and produced a slower, dearer gate with no evidence behind it.
§6What another builder should take from this
None of these four fixes were hard. Together they were an afternoon, and the code has been in production since 22 June. What was hard was the thing that preceded them, which was sitting down with a literature that mostly gets cited to justify building a judge, and using it instead to interrogate the judge we had already built and already trusted.
If you run an LLM as a gate, the question is not whether it works. It is which known failure mode you have left switched on, and whether the number you use to reassure yourself is a real one.
The one to check first is the cheapest and the most likely: are you collecting a signal from your model and then throwing it away at the decision? We were, for weeks, in the one place it counted.
Is a panel of LLM judges better than one big judge?
The research says several small judges from different model families can beat a single larger one, mainly because their failure modes do not correlate, and it is far cheaper. Our three-juror panel costs about $0.002 per candidate. The caveat we act on is that the replication evidence thins out as you add jurors, so we run three lineages and stop rather than treating a bigger panel as automatically better.
How should a judge panel break a split decision?
If your jurors report confidence, use it. Counting heads throws away the strongest signal you have: a majority that is unsure, against a minority that is certain, should not automatically win. We sum each side's confidence and take the higher, and treat a near dead-heat as a kill, because a false build costs more than a missed one.
Why is agreement rate a bad metric for an LLM judge?
Because it rewards guessing the base rate. On a set that is 90% PASS, a judge that always says PASS scores 90% agreement while being useless. Cohen's kappa corrects for chance agreement, which is why we track it instead, with a hard rule to halt automation below 0.60.
vera/poll.py and changed four things: confidence-weighted resolution of split votes, rationale-before-verdict in the required JSON schema, an explicit anti-verbosity and anti-surface-bias instruction in the juror system prompt, and Cohen's kappa in place of raw agreement rate as the calibration measure (vera/calibrate_kappa.py). The changes shipped and were tested on 22 June 2026; the pre-audit file is preserved at poll.py.bak-20260622-vera-harden and the code excerpts above are quoted from that diff. Unanimous verdicts and the human-escalation path were deliberately left untouched. We report no kappa figure because our labelled set is six rows, which is too small to support one.