Our nightly eval writes a fail with a reason every time an agent's output falls below bar, and we had 303 of them stacking up as separate tickets, the same problems re-reported night after night. We built the step of the self-improvement loop everyone sketches and nobody shows: read the failures, cluster them by who failed and how, and distil each cluster into one root cause. The 303 collapsed into five. And 76 per cent of them turned out to be our own logging, not our agents.
What we did
The tool reads the kill verdicts out of the eval's own score log (2,672 graded outputs, 303 of them failed), groups them by which agent action they came from, and hands each cluster to a model with one job: find the single recurring root cause, say whose fault it is, agent or rubric or logging, and write one concrete fix. A model fallback means a flaky call never silently drops a cluster. What comes out is not 303 tickets. It is a short, ranked list where each line covers dozens of failures at once.
Why it was worth doing
The split is the story. Of the clustered failures, 221 were logging: the eval was grading a truncated preview of the output, so real fields looked missing and complete rationales looked cut off mid-sentence. Sixty were genuine agent faults, small and real, a null field here, the wrong category there, a list passed as a Python string instead of JSON. Nine were a stale rubric failing correct work. Pointed honestly at our own flags, the self-improvement loop did not tell us to improve the agents. It told us to fix the eval.
This lands harder because of what we shipped earlier the same day. This morning we fixed exactly one of these logging clusters, a juror whose rationale was being clipped before the grader saw it. We thought it was a one-off. The distiller shows the identical bug across four different agent actions, 221 failures in total, most of them still live tonight. One fix at the logging layer, grade the full output and not the preview, clears three quarters of the board. We would not have known the ceiling was that high without collapsing the pile first.
What's still off
Be honest about what this is. The distillation is a model reading reasons, so a cluster's root cause is a strong hypothesis, not a proven diagnosis; you still read it and check before you cut code. It clusters by agent action, so a pattern that cuts across actions, like this logging one, only becomes visible when you read the fault column, not from the grouping itself. And it distils, it does not fix. Turning "grade the full output" into a landed change across four actions is the next ship, teed up but not done. This is the cheap half of the loop, and it is worth saying plainly: reading a failure is cheap, and the cure is still the work.
What's now in the stack
vera/distill.py: reads the eval's kill verdicts, clusters by agent action, distils each into a root-cause fix tagged agent, rubric or logging.- A standing finding: about three quarters of our eval failures are logging artifacts, one bug class, not agent quality.
- A ranked fix list where the top line, a logging-layer change, retires the majority of the flag backlog.
- A standalone copy on GitHub. Steal what you like.