On a real-world benchmark, an AI agent wrote code that worked 61% of the time and was secure just 10.5% of the time. It runs and is still insecure nine times out of ten, and you cannot prompt your way out of it: giving the agent security guidance barely moved that number and made the code less correct. The only thing that reliably catches this is a reviewer that is not the model that wrote the code. We already run a panel of AI jurors over our own work, but a panel of one bloodline shares blind spots and can wave a bug through unanimously, so we added a juror from a different bloodline. Claude wrote the code and reviewed it; a GPT-family model reviewed the same diff blind; the tool shows you only where they split. On a real change, both caught a SQL injection and an integer overflow, and Claude alone caught three more. That gap, not the agreement, is the whole product.
The number behind it
On SusVibes, a benchmark that has a coding agent solve real-world tasks and then checks the result for vulnerabilities, an agent driving Claude Sonnet produced code that was functionally correct 61% of the time and secure only 10.5% of the time. The two numbers are almost unrelated: the code works and is still exploitable nine times in ten. And the authors tried the obvious fix, telling the agent to write secure code and even handing it the class of vulnerability to avoid, which barely improved security and dropped correctness by about six points. You cannot prompt the blind spot away. The only lever left is an independent check by something that did not write the code, and did not inherit the reasoning that produced the hole. That is the entire job of this tool.
The panel that agrees with itself
We have a thing called Vera: a panel of language models that scores the fleet's own output against a rubric, votes, and gates the work that falls short. Panels are good. They beat a single judge on almost every measure.
They have one failure that does not show up until it bites you. If the jurors are the same model family with different prompts, they do not fail independently. They share the blind spots that came with the training. So the panel can return a clean, confident, unanimous verdict and be unanimously wrong, and the confidence reads as signal when it is actually just three copies of the same mistake. Agreement across a panel is only worth as much as the independence of the people in it. Same bloodline, correlated errors, and the vote count lies to you.
That is not a hypothetical we invented for the write-up. It is the exact thing the research on LLM-judge panels warns about, and it is why "we added more jurors" is not automatically "we got a better answer".
So we added a juror from a different bloodline
The fix is not more jurors. It is a juror that fails differently. Claude builds the code and reviews it, because it is right there and cheap. Then the same diff goes to a GPT-family model, cold: no memory of writing it, no access to the reasoning, read-only. Both reviews are forced through the same strict schema, so a finding that cannot state a severity, a real confidence and a reason does not get to vote.
Then the tool sorts the findings into three buckets and that is all it does:
- Both lineages raised it. Trust it, act. Two different training histories arrived at the same defect independently, which is about as strong as a review signal gets.
- Only Claude raised it. Either Codex missed a real bug, or Claude is over-flagging. A human glance settles it in seconds.
- Only Codex raised it. Claude's blind spot, or Codex is over-flagging. Same glance.
The agreement is the part you can stop thinking about. The disagreement is the part you look at. That inversion is the whole idea.
What it looked like on real code
Last night we pointed it at a seeded file and let both lineages run. The output, verbatim in shape:
AGREED (2): trust these, both lineages raised them
[HIGH] SQL injection via f-string interpolation into a raw query
[MEDIUM] Integer overflow / silent truncation computing total_price
DISAGREEMENT is the signal below:
CLAUDE ONLY (3):
[HIGH] (0.80) Database connection never closed on the exception path
[MEDIUM] (0.75) Race condition between stock check and stock decrement
[MEDIUM] (0.72) Sensitive order data logged at INFO level
CODEX ONLY (0)Read it the way it is meant to be read. The two AGREED findings need no argument: two bloodlines, same bug, done. The three CLAUDE-ONLY findings are the actual work of the review, because a leaked connection, a stock race and PII in the logs are all real, and Codex said nothing about any of them on this pass. And CODEX-ONLY being empty here does not mean the second lineage is dead weight. On other diffs it is the one that speaks up when Claude has gone quiet. You do not know in advance which model is going to be the one that sees the thing, which is the entire reason for running two.
Why we do not average it away
Here is where this parts company with a normal panel. A panel exists to collapse many opinions into one verdict, usually by vote. If you did that here, majority rule would take those three CLAUDE-ONLY findings, note that only one of two reviewers raised them, file them as the minority view, and drop them. You would have voted away three real bugs to feel more confident about a number.
For bug-finding that is exactly backwards. A defect that one competent reviewer sees and another misses is still a defect. It is not less true for being outvoted. Disagreement between two capable, independent reviewers is not noise to be smoothed out, it is a map of where each one is blind. So the tool refuses to average. It keeps the split and hands it over. The judge is not the vote; the judge is you, reading the short list of things the two models could not agree on.
What we are not claiming
The honest boundaries, because a review tool that oversells itself is worse than none:
- Agreement is strong, not proof. Two bloodlines can still share a blind spot, just far less often than one lineage consulted twice. AGREED means "very likely real", not "certainly real".
- It is non-deterministic. Run it again and a finding can move between buckets, a CLAUDE-ONLY on one pass becoming AGREED on the next. A single run is a floor on what is wrong, never a ceiling.
- The matcher is a heuristic. Deciding that two findings describe the same bug is done by file and overlapping keywords, not deep understanding, so it can occasionally split one bug into two or pair two that are not the same. It is good enough to be useful and honest enough to say it is a heuristic.
- It costs real money and minutes. A cross-lineage run is tokens and about half a minute, versus nothing for a second pass of the model that already wrote the code. So it belongs on diffs where being wrong is expensive: money, auth, data loss. Not on a formatting change.
What is in the stack
-
disagree.py, a standalone, dependency-free version, is on GitHub. It reuses the blind Codex reviewer from the previous ship and adds the Claude reviewer and the reconcile step. Steal what you like. - The lesson feeds back into Vera. A panel of same-family jurors is a panel with correlated blind spots, and the strongest fix is not more of them, it is one juror from a different bloodline. Disagreement between bloodlines is worth more than agreement within one.
FAQ
Why is a panel of AI models sometimes confidently wrong?
Because if the jurors share a training lineage they share blind spots, so they fail on the same inputs together. A unanimous, high-confidence verdict from three near-identical models is not three independent confirmations, it is one mistake counted three times. The strength of a panel comes from the independence of its members, not their number, so a cross-lineage juror that fails differently is worth more than another same-family one.
What is a cross-lineage code review?
Two models from different training families review the same code independently, and you compare the results. Findings both raise are very likely real. Findings only one raises are where the two disagree, which is the part worth a human's attention, because it is either a real bug the other model was blind to or a false alarm you can dismiss in seconds. The value is entirely in the two models failing in different ways.
Should you average or vote away a finding only one reviewer raised?
For bug-finding, no. A defect that one competent, independent reviewer sees and another misses is still a defect, and voting it out to raise your confidence score throws away real bugs. Disagreement between two capable reviewers is a map of where each is blind, not noise to smooth over, so surface it and judge it rather than letting a majority rule bury it.