We reproduced DarwinX, a paper that evolves the agent harness (the scaffolding around a model: retries, temperature, self-check, task decomposition, tool order) with a genetic algorithm instead of tuning it by hand. The good news: the evolved harness beat our sensible hand-tuned default by 14.6%. The honest news: it beat plain random search, at the same evaluation budget, by only 3.1%. So we dug into when evolution actually earns its keep. The answer is not what we expected.
What we did
We built a small, seeded, GPU-free reproduction. A harness is a genome of six genes (retry count, temperature, self-check on or off, decompose on or off, reflection depth, tool order). Fitness is the weighted success rate across a battery of six task types, each of which rewards a different setting: flaky-tool tasks reward retries, long-horizon tasks reward decomposition, precise tasks punish high temperature, and so on. So no single fixed harness is optimal for the whole mix, which is exactly the case where you would hope evolution helps.
Then we ran a genetic algorithm (tournament selection, one-point crossover, mutation, elitism) for 480 evaluations, and gave plain random search the same 480 evaluations as a control. That control is the part most evolutionary-agent write-ups quietly skip, and it is the part that matters.
Why it was worth doing
The headline result is real: evolving the harness beats a reasonable default by 14.6%, with per-task gains everywhere. If you are hand-picking your agent's retry and decomposition settings, you are probably leaving that on the table.
But random search sat only 3.1% behind the genetic algorithm, because our harness space is just 720 configurations and 480 random draws nearly brute-force it. So we grew the space with extra knobs, expecting evolution to pull ahead. It did not. Padding the space with knobs that do not affect the outcome helped neither method and slightly hurt evolution, which wasted mutations on dead genes. Size was not the driver.
The driver is epistasis: how much the genes interact. We tested it on a clean, tunable NK landscape (N=20 genes, K interactions per gene), same eval budget for both methods, averaged over eight seeds.
At K=0 (independent knobs) evolution wins a little. At K=2 (moderate interaction) its margin over random search peaks at 7.8%. At K=12 (near-random ruggedness) it collapses to 0.2%, because there are no stable building blocks for crossover to recombine, so the genetic algorithm degrades to random sampling. Selection earns its keep in the middle, where the harness knobs interact but the landscape is not chaos.
What's still off
This is a simulated executor, not a live model bake-off. The genes behave as those knobs behave in a real loop, and the evolutionary dynamics are the object under test, but the absolute success numbers are synthetic by design. The point is the shape of the result, not the decimals, and the shape is robust across seeds.
The practical takeaway we will actually use: before building an evolutionary rig to tune an agent harness, run random search first. It is simpler and often within a few percent. Reach for evolution only when your harness knobs genuinely interact, which is the regime where it is worth the extra machinery. Everything here (code, seeds, all three experiments) is in the mirror below and reproduces from one seed.