Our nightly prompt optimiser (GEPA) had been promoting luck. It staged 35 "winning" prompts for review since June, each claiming 5 to 22.5 points over the live prompt. Re-run today, all six of the most recent winners score below the prompt they supposedly beat. The cause: the same prompt, run three times, swings by 7.5 points, and the bar to win was 3. We have added a held-out confirmation step, and switched off a second optimiser that had done nothing since May.
What GEPA does
GEPA rewrites one of our agent prompts every night. It makes five variants (shorter, an added example, reordered rules and so on), scores each against 40 labelled test items, and stages any variant that beats the current prompt by 3 points for a human to approve. The prompt it works on today is the Loop triage editor, which sorts incoming research into note, news, ship or skip.
What we found
A paper on self-evolving models (EvoHeir, arXiv 2610.22029) names the flaw: if you choose the winner and measure the win on the same questions, the win looks bigger than it is. Most "improvements" do not hold on fresh questions. So we tested our own loop.
- Noise. The current prompt, run three times on the same 40 items: 97.5, 95.0 and 90.0. Last night's run scored it 92.5. A 3 point bar sits inside that spread.
- Replays. We re-ran the six most recent staged winners three times each. All six average below the current prompt's 94.2. The cleanest case, staged on 18 October with a claimed +5, averages 90.8, a 3.3 point loss. (The other five were staged when the live prompt scored around 60 on an older model route, so 18 October is the only like-for-like comparison.)
- Nobody noticed because nothing auto-promotes. 35 candidates piled up unreviewed. The human gate held, but it was reviewing a list of coin flips.
What we changed
The 40 items now split into 28 selection rows and 12 confirmation rows. The split is fixed by a hash of each item, so a row never changes side, and it keeps the same mix of answers in both halves. Variants are ranked on the 28. A would-be winner must then also beat the current prompt on the 12 it was never chosen on, or it is not staged. Every row is still scored once, so it costs nothing extra.
The first run showed why it matters. Two variants tied for first on selection. On the unseen 12, one gained 8 points and the other lost 8. The old code could not tell them apart.
What's still off
Twelve rows is small. One item is worth 8 points, so the confirmation step catches flukes but cannot make a win trustworthy. The variant that passed tonight is the same kind of edit as the one that failed its replay. The real fix is a larger test set and a bar set above the measured noise. Until then, nothing GEPA stages gets promoted.
Our other target, the skill distiller, has a single test item, so it cannot be split. It now stages winners marked unconfirmed rather than pretending.
What's now in the stack
gepa.py: deterministic, stratified 70/30 split, ranking on selection rows, held-out confirmation before staging, and both deltas logged on every candidate.evaluator.py: results now come back in row order, which the split depends on.experiments/replay_winners.py: re-scores past winners on fresh runs, for an honest look at any optimiser's track record.- Maggie's v0 optimiser, baseline-only with zero sends since May, is retired from cron.