Agents relearn the same fix on every run. Correct a model's reasoning today and, unless you paste that correction back into context tomorrow, it takes the same wrong turn again, so the fix becomes a line you re-type forever, growing the prompt each time. We built deepfix: you edit the one reasoning step that went wrong, then distil that edit into a short standing rule that prevents the whole class of error on later runs. In our test a 114-token worked correction distilled to a 25-token rule, 78% fewer tokens, saved on every run for the life of the agent.
What we built
deepfix is four small verbs over a captured reasoning trace.
edit records a human rewrite of one step: the point where the
model reasoned its way somewhere wrong. distil turns that
one-off correction into a compact, reusable directive. apply
composes the next run's prompt with the accumulated directives prepended.
cost counts what the fix costs if you carry it inline on every
run versus injecting the distilled rule.
The honest part is distil. Compression is the model's job, not
a regex's, so without a model deepfix stores the correction verbatim and
says so out loud: no model, no compression, no saving. Give it a model and
it writes the general rule that would have prevented the error, and only
then does the token maths turn in your favour. We would rather the tool
admit the offline path saves nothing than pretend a passthrough is a
distillation.
What the demo showed
The scenario is real and comes from our own fleet. An agent is given a
global rule, never use any, then asked to write a
markdown plan. It burns a whole reasoning step trying to enforce a
TypeScript rule on prose, where it does not apply. A human edits that one
step. deepfix sends the wrong step and the human fix to the model, which
distils them into a rule: "Enforce code rules only on source code files
(.ts and .tsx), not on documentation, comments, or prose content."
Then the token maths. Carrying a short one-line fix inline costs 39 tokens
a run and the rule costs 25, a modest 36% saving. Carry a realistic worked
correction, the kind you actually paste to re-teach reliably, and it costs
114 tokens a run against the same 25, a 78% saving, 8,900 tokens over a
hundred runs. The directive stays around 25 tokens no matter how long the
original correction was, so the more careful your re-teaching, the more
distilling it saves, and it saves that on every run from then on. Counts
are from tiktoken, not estimates.
Why it was worth doing
Corrections should compound. An agent you fixed once should get permanently better at that class of thing, cheaply. Most setups do neither: re-running the task forgets the fix and pays full price again, and pasting the worked correction into the system prompt keeps the fix but bloats every future call and stacks up as more corrections arrive. Distilling to a rule is the cheap middle, a short standing instruction that carries the lesson without carrying the story. It also matches how a person learns from a mistake: you keep the rule, not the transcript.
What's now in the stack
deepfix.py—edit,distil(with--llm),apply,cost. Stdlib only; token counts via tiktoken when present, a flagged characters/4 estimate otherwise.demo.py— the whole loop end to end, offline and deterministic by default,--llmfor real distillation against Claude.test_deepfix.py— eight offline tests, no network, no model calls.- The full source on GitHub, with the write-up in examples/104. Steal what you like.
What's still off
The distillation is only as good as the model that writes it: a lazy model writes a lazy rule, and a rule can over-generalise, so read the directive before you trust it. Directives accumulate and nothing here dedupes or retires a stale one yet, which is the obvious next piece. This is the interaction mechanism, editing the reasoning and keeping the lesson as a prompt; it does not touch weights and is not a training loop. And it is running on our own tooling only for now, which is where anything that edits how our agents reason should earn its keep first.
FAQ
How do you stop an agent making the same reasoning mistake every run?
Correct the reasoning step that went wrong, then distil that correction into a short standing directive and inject it on later runs, rather than re-running the task from scratch or re-pasting the full worked correction each time. The directive is a compressed rule, so it prevents the whole class of error for a fraction of the tokens.
Does correcting an LLM's chain of thought reduce token usage?
It can, if you distil the correction. Carrying a full worked correction inline on every run costs its full length every time. Distilling it once into a short rule costs roughly a fixed 25 tokens regardless of how long the original correction was. In our test a 114-token correction became a 25-token rule, a 78% saving repeated on every run.
Why not just add the correction to the system prompt?
You can, and a distilled directive is exactly what you would want to add. The failure mode is adding the raw correction instead of a distilled rule: worked examples are long, they accumulate, and the system prompt grows without bound. Distilling first keeps the standing instruction short and general.