To tell whether a change to an agent helped, we run the old version and the new one over the same set of test scenarios and score both with a panel of three LLM judges. It works. It is also wasteful, and the waste has an exact shape: most of those judgements are spent on scenarios where the two versions give the same answer. Those are ties. A tie cannot tell you which version is better. We were convening the whole panel to rubber-stamp them anyway.
Where the number you care about actually comes from
An A/B eval produces one number that matters: the net change in pass rate between the old version and the new one. Walk through where that number comes from, scenario by scenario. If both versions pass a scenario, it is a stable-pass. If both fail, a stable-kill. Neither can count as fixed or regressed, so neither moves the net delta by a single point. The delta is built entirely from the scenarios where the two versions disagree: one passes, the other fails. Everything else is a tie that nets to zero. Paying the full three-juror panel to confirm a tie buys you no information about which version won. It is the eval equivalent of recounting the votes in a district both candidates conceded.
What we built
coeval is a drop-in alternative to our A/B harness that spends
the expensive panel only where the two versions disagree. The method is
lifted from a recent paper, Task-CoEvolve, which calls it variance-weighted
sampling near the capability frontier: put your evaluation budget on the
cases that actually separate the candidates, and estimate the result from
that subset. We took the idea, not the paper's cloud harness, and wired it
into the eval stack we already run.
It works in two passes. First it generates both versions' responses for every
scenario, then screens each one with a single cheap judge
instead of the full panel, at about a third of the cost. That gives a
provisional verdict and a confidence for each side. Then it scores every
scenario by how much it can separate the two versions: a
frontier weight of 1.0 when the two cheap screens
disagree, since only those scenarios can move the delta, and
otherwise 1 - min(confidence), so a shaky agreement outranks a
settled one. It escalates to the full three-juror panel every disagreement,
plus every agreement the screen was not confident about, and lets the
confident ties keep their cheap verdict. Same net-delta report as before, a
fraction of the judging.
What it costs, and where the saving is
The saving is not magic and it is worth being precise about. A scenario that
gets escalated actually costs more under coeval,
because it pays for the cheap screen and then the full panel on top. All of
the saving comes from the ties you skip: on those you pay one cheap screen per
side and stop. So the cheaper your change is to tell apart, the less you save,
and the more your two versions agree, the more you save. On our live demo set,
where the two versions agreed on most scenarios, it panelled two of four and
came in between a third and two-thirds cheaper than judging all four, run to
run. Point it at a hundred scenarios where a prompt tweak only shifts a
handful, and you are judging the handful, not the hundred.
The cost accounting is grounded in the run's own measured prices, not a guess: it prices the skipped panels at what this run's panels actually cost, so the reported saving is real money, not a model of it.
Proving it without trusting a screenshot
The behaviour that matters is a selection rule and some arithmetic, so we pinned it down with tests that call no models at all: feed in a set with known disagreements and confident ties, inject fake screen and panel verdicts, and assert that every disagreement is panelled, every confident tie is not, the net delta matches what a full panel would have found, and the cost and saving add up to the penny. Eight tests, deterministic, no tokens spent. The live demo then runs the same code against our real scenario set with the real judges, so the integration is exercised end to end and not just the maths.
What it does not do
The cheap screen is a single judge, so it has no redundancy: a scenario it
rates a confident tie could hide a flip the full panel would have caught. That
is the one real risk, a missed frontier, and we did not paper over it. Two
guards hold the line. A confidence floor decides how much benefit of the doubt
a tie has to earn before it is allowed to skip the panel, so you can dial the
saving down and the safety up. And a screen that errors is never trusted as a
tie; it is always escalated. When you genuinely need the exhaustive ground
truth, you turn the frontier off and panel everything, which is just the old
harness. The point of coeval is that you rarely need to. The
cases where two versions of anything agree are not where the truth is in
doubt. Spend the expensive judge where the answer actually forks.
The harness, the tests, and the live demo are on our GitHub mirror.