A verification paper was doing the rounds, and its idea was good enough that we wanted it in our verifier. Instead of sampling more answers, break a reasoning trace into its decision-critical claims and hunt each one for a single decisive flaw. So we built it, ran it against the plain one-pass check we already had, and it lost. It caught fewer than half the bugs the one-liner caught, raised false alarms the one-liner did not, and cost twice the tokens. The reason it lost turned out to be worth more than a win.
The bake-off
Fourteen reasoning traces, seven with exactly one decisive flaw and seven genuinely sound, with the ground truth fixed by hand so no model graded anything. Two verifiers, the same model underneath, temperature zero. The first is holistic: one call, here is the whole trace, is it sound or flawed. The second is the claim-level method: decompose the trace into its critical claims, then give each claim to an isolated call that sees only the problem and that one claim and is told to find a decisive flaw. If any claim is refuted, the trace is flawed.
The isolation is meant to be the strength. A refuter chasing one claim cannot be lulled by the confident prose around it. That is the theory. Here is what happened.
| verifier | recall on flawed | accuracy | false alarms | tokens |
|---|---|---|---|---|
| holistic (one call) | 1.00 | 1.00 | 0 | 6,527 |
| claim-level | 0.43 | 0.57 | 2 | 13,589 |
The one-liner caught every flaw, raised no false alarm, and did it on half the tokens. The clever pipeline missed four of seven flaws, cried wolf on two sound traces, and cost twice as much.
You can't refute a claim you never extracted
The failure is not random, and it is not the model being weak. It is the step we did cheaply. Look at what the decomposition actually pulled out of the traces it then missed. On a trace whose only flaw was the arithmetic "15% of 24 is 4.60", the claims it extracted were about the base price and the discount applying, both true. The wrong number never became a claim. On a trace that computed "150 divided by 2.5 is 55", it extracted "average speed is distance divided by time", true, and a stray fragment. The actual wrong division never became a claim. On an invalid syllogism, it extracted both true premises and left the fallacious leap between them unstated.
So the isolated refuter did its job perfectly and found nothing, because it was handed only the true parts. You cannot refute a claim you never extracted. The whole method rests on a decomposition step that has to surface the load-bearing wrong move as something checkable, and when it surfaces the sound premises instead, the flaw walks through the gap between the claims untouched. Meanwhile the holistic reader, seeing the arithmetic and the answer in one view, just noticed the number was wrong.
It also cried wolf
The false alarms are the same coin. A refuter told to find a decisive flaw and not to be charitable will, on a genuinely correct claim, sometimes talk itself into one. Isolation makes that worse, not better, because it strips the context that would have reassured a reader the step was fine. Two sound traces got flagged as flawed for exactly this reason. A verifier that both misses real bugs and invents fake ones is worse than the plain check on both axes at once.
This is not the paper being wrong
The paper is not wrong, and the point here is not to dunk on it. Its wins come from careful claim extraction and from aggregating evidence across many samples on genuinely hard problems, on which a single holistic pass does struggle. What we built was the naive port: it kept the expensive shape, one call per claim, and threw away the part that made the method work, the quality of the extraction. The shape is the cheap bit to copy and the quality is the whole product. That does not transfer for free, and a benchmark headline will never tell you it did not.
What's still off
This is a spike, not a verdict on the technique. Fourteen traces, one model, one flaw apiece, hand-authored problems, so it measures the naive port and nothing grander. A fair rematch would need a claim extractor tuned to surface computational steps as checkable claims, a refuter that is not browbeaten into false positives, and aggregation rather than first-refutation-wins. We may run that. But the transferable lesson holds regardless, and it is the boring one that keeps being true: before you swap a plain verifier for a fashionable pipeline, measure it on your own traces, because the unglamorous step you skipped is usually the one doing the work.