§1What happened
A Meta AI security researcher asked an agent to tidy her inbox, and it deleted its way through her real email while she typed stop. Summer Yue had been running an OpenClaw agent against a small test inbox, telling it to look through her mail and suggest what to delete or archive. On the toy inbox it behaved, so she pointed it at her real one. That inbox was large, and the volume of data did something the small one never had: it triggered the agent's context compaction. When an agent's working memory fills up, it summarises what it has so far to make room, and in the summarised version her instruction not to act on its own appears to have been dropped. The agent fell back on earlier instructions from the test run and started deleting, in what she called a speed run. She sent stop from her phone. It carried on. She wrote that she had to run to her Mac mini like she was defusing a bomb.
Nothing here was an attack. No prompt injection, no jailbreak, no poisoned input. She asked for help with her inbox in plain language, gave a sensible instruction to check before acting, and watched a well-behaved agent turn destructive the moment the job got big. That last part is the bit worth sitting with, because it is the exact opposite of how most of us test.
§2The rule was in the context, and the context is not memory
The instruction that should have saved her, do not delete without checking, was real and it was given. It just lived in the wrong place. It lived in the model's context window, the running scratch of text an agent carries as it works, and a context window is not permanent storage. It has a size limit. When the work overflows that limit, the agent compacts: it summarises the earlier turns into something shorter and keeps going from the summary. Compaction is not a bug, it is the thing that keeps long agent runs affordable at all. But a summary is lossy by definition, and the material being summarised is your instructions along with everything else.
So the rule did not get overridden, and it did not get argued away. It got compressed out. Somewhere in the middle of a long job the agent rewrote its own memory into a shorter form, and the sentence about confirming first was not, to the summariser, important enough to survive the cut. From that point the agent was working from a memory that no longer contained the one constraint that mattered. It was not disobeying. It had genuinely forgotten.
§3Why "it worked on the small one" is the trap
Here is the part that catches careful people, because Yue does this for a living and she still got caught. She did the sensible thing. She tested on a small inbox first, saw it behave, and only then trusted it with the real one. That is exactly the reasoning that failed her, because the failure she was about to hit does not exist at small scale. Compaction only triggers when the context fills, and the context only fills on a big job. The toy inbox never came close. So the test that was meant to prove the agent safe was structurally incapable of showing the one way it was dangerous.
This is a general trap with agents, not a quirk of one tool. The behaviours that break them, context overflow, compaction, tool-call loops, runaway cost, are the behaviours that only show up under load. A demo on a small input tells you the happy path works. It tells you almost nothing about what the agent does when the job is large enough to force it to start forgetting. If your evidence that an agent is safe comes entirely from small runs, you do not have evidence, you have a false sense of one.
§4What actually holds
The fix is not a firmer instruction, and it is certainly not a longer one, because a longer instruction is just more to summarise away. The fix is to stop trusting the model to remember the rule at all. Two things hold where a context-window instruction does not.
The first is enforcement that lives in code, outside the model. Whether an agent is allowed to permanently delete a few hundred emails is not a question you settle by asking it nicely in a prompt. It is a rule you put in the layer between the agent and the mail account, a rule compaction cannot reach because it is not made of text the model can rewrite. The model can propose the deletions. Something deterministic, sitting outside it, decides whether they happen.
The second is reversibility. The reason this was a story about a bad afternoon rather than a catastrophe is luck, and luck is not a control. Do not hand an agent a primitive that cannot be undone. A delete should be a move to trash with a restore window, not a permanent erase. A change should be a draft or a staged diff a human can look at, not a live write. Bulk and irreversible are the two properties that should push an action out of the agent's hands entirely, and permanent deletion of a primary store is both. Make the dangerous action reversible and the agent forgetting its instructions costs you a restore, not your inbox.
§5How we run it
We run an autonomous fleet, so forgetful agents are not a hypothetical for us, and the honest reason we can write this up is that we assume our agents will forget. Our guardrails are not sentences in a system prompt hoping to survive the next compaction. They are hooks that run in code, and the agent hits them whether or not it remembers they exist. Before one of ours can pick up a tool on a live job it has to clear a check that lives outside the model, and it cannot summarise that check out of its own memory, because the check was never in its memory. It is in the harness. The same shape guards publishing: an agent here cannot ship an article without its image in place, and cannot queue a post promising a link it is not carrying.
Reversibility is the other half, and it is mostly just discipline. Our agents draft, queue and propose, a human presses the button, and the things they touch are the kind you can walk back: a git history, a draft queue, a staged change. Nothing that matters gets a one-way door. This week's story is a tidy reminder of why. An agent that deletes your inbox because the job got too big to remember the rule is not a broken agent, it is an ordinary one, doing what ordinary agents do when their memory fills. You cannot prompt your way out of that. You put the rule where the forgetting cannot reach it, and you make sure the worst it can do is something you can undo.