There is a recent paper, Harness Handbook (arXiv:2604.25850), that argues an agent edits its own code better when it reads a map organised by behaviour instead of a file tree. We reproduced it and pointed it at one of our own agents. The claim held: change-site localisation F1 went from 0.83 to 0.97, and edit-plan quality from 3.7 to 4.6 out of 5. The load-bearing part is not the prose, it is the bit that refuses to write down anything it cannot prove.
What we did
We built the paper's three-stage pipeline and ran it against
maggie, our outbound-email agent (7 files, 1,582 lines, 43
functions). Stage one is a deterministic pass over the code with Python's
AST: it produces a program graph of every file, function, call edge and
module-level bit of state, each carrying a file:line span.
Stage two asks a model to reorganise that graph by behaviour, not
by file: a single request traced end to end, then eight behaviour units
(batch send, the pre-send verifier gate, the AP2 mandate signer, the
cadence engine, the inbox watcher, and so on). A reviewer pass then attaches
code evidence to each unit.
Stage three is the one that matters. Before any evidence link is written into the handbook, it is checked against the Stage-one index: if the cited symbol does not resolve to a real file and line range, it is dropped. On this run, 39 of 39 links resolved. The output is a handbook where every claim about what the code does points at code that exists, plus an L3 detail per unit covering the trigger, the permission rule, the state it changes, the success and failure paths, and the edge cases.
Why it was worth doing
We did not want to take the paper's word for it, so we ran an A/B. Four cross-module change requests, two repeats each, same coding agent (Claude Sonnet 4.6). One arm got a plain repo map (file tree, docstrings, function signatures). The other got the generated handbook. We scored where each arm said the change belonged as file-level precision, recall and F1 against hand-verified gold sites, and had two independent, blinded judges rate the edit plans one to five.
- Localisation F1: 0.83 → 0.97.
- Recall: 0.88 → 1.00. The handbook arm never missed a change-site.
- Edit-plan quality: 3.69 → 4.56 out of 5.
The example that shows why: asked to make the follow-up gap configurable
instead of hard-coded at three days, the baseline found cadence.py
(and once a spurious db.py) but missed sender.py,
where the first gap is actually set after an initial send. It would
have shipped a half-fix. The handbook arm found both, every time, because
the L3 state-change field for the send unit names that write out loud. That
is the whole thesis in one diff: a behaviour map surfaces a dependency a
file-tree skim walks straight past.
What's still off
This is one harness, four tasks, eight trials. It is directional, not a benchmark. The coding agent and one of the two judges share a model family, which we softened with blinding and a second-family judge but did not remove. And the gold change-sites are file-level and verified by us, not an external oracle. The honest takeaway is narrower than the headline number: the deterministic evidence gate is what turns a confident-sounding summary into something an agent can trust, and that gate is cheap to build. We would want a bigger spread of harnesses before promising the F1 lift to anyone.