A long-running agent keeps a block of hard rules at the top of its context, then works for hours. When the tool compacts that context to save tokens, the rules get summarised along with everything else. We measured what survives. All 22 of our test rules were gone, word for word, after the very first compaction. The agent kept the gist of the obvious ones, so nothing looked wrong. The two rules it could not guess from common sense were silently replaced with confident, wrong answers. Pinning the rules in a block that is never summarised held all 22, at no extra cost.
What we measured
We built a small harness, compaction_cliff.py. It gives an agent
a policy of 22 hard rules, the kind you never want lost: fail open on a cost
error, never send marketing at the weekend, escalate a low-confidence call to
the compliance queue. Then it buries them under a long, realistic transcript
of tool calls and chatter. Each round it asks a real model (Llama 3.3 70B, via
Together) to compact the transcript with a plain "summarise this so you can
keep working" instruction. Nothing is rigged to drop the rules. After each
round we check two things: does each rule survive word for word, and does the
agent still behave the way the rule requires.
The verbatim number falls off a cliff at once. Round one: 0 of 22 rules survive intact, and it stays at zero for every round after. The summariser paraphrases everything, because paraphrasing is the job.
baseline verbatim curve: [22, 0, 0, 0, 0, 0, 0]
triage verbatim curve: [22, 22, 22, 22, 22, 22, 22]Behaviour is subtler, and more dangerous. Five of seven probes still passed, because a capable model can rebuild an obvious rule from common sense: asked whether to send a campaign on a Saturday, it says no, even with the rule long gone. That is exactly why you never notice the loss. The two rules it could not reconstruct, an arbitrary code word every export must carry and the specific queue escalations go to, failed. The agent did not say it was unsure. It invented answers: a made-up tag, and "the open support tickets queue". Confident, plausible, wrong, and no record that a rule was ever there.
Why it matters
Compaction is not a rare event on a long job. It happens again and again, and the research that named this problem ("Knowledge Triage") found that standard compaction keeps only a fraction of safety rules after a few rounds. Our run is blunter: exact wording is gone on the first pass. For most rules that is survivable, because the model's defaults happen to line up with what you wrote. The problem is the rule where they do not: a threshold number, a "fail open, not closed" direction, a named exception, a code word. Those are exactly the rules you bothered to write down, because they are not guessable. They are the first to vanish and the last you would notice, because the agent keeps answering, just wrongly.
The fix is boring and total. Keep the policy out of the summariser. We pinned the 22 rules in a separate block that is passed through untouched every round. Verbatim retention went to 22 of 22, every round; behaviour to 7 of 7. It costs nothing extra, because the rules are already text you were carrying. You just stop feeding them to the compressor. That is per-type retention: some context is a conversation you can summarise, some is a contract you must keep word for word, and the two should not share a fate.
What's still off
This is one open 70B model doing the compacting, not every model, and our transcript is synthetic. A stronger summariser might hold a rule's meaning longer, and a longer real job would bury the rules deeper. We measured verbatim survival and a handful of behaviour probes, not every way a paraphrased rule can bite. And "pin the rules" is easy when you know which text is load-bearing, harder when your policy is scattered through a sprawling system prompt. The headline holds either way: if a rule's exact wording is the safety property, do not let it near a summariser.
What's now in the stack
compaction_cliff.py: runs real compaction rounds over a rule set and measures both verbatim and behavioural survival, baseline against pinned.- A 22-rule demo policy and seven behaviour probes, two of them rules the model cannot guess, which is the decisive test.
- The full run log and
result.jsonfrom the run in this write-up. - A standalone copy on GitHub. Steal what you like.