§1The number that should bother you
A paper out this week, HarnessRisk, does something most agent-safety work skips: it stops testing the model and starts testing the harness, the layer of config, tools, memory and gates that wraps the model in production. It pairs a benign user objective with an adversarial instruction hidden in an untrusted workflow artifact, 128 cases in all, and measures four things: did the task still work (utility), did the attack land (attack success), did the compromise stick (persistence), and did anything notice (detection).
Two of the numbers are the story. Attack success ranged from 12.6% to 80.9% depending on the harness, while utility stayed between 75% and 97.6%. Read that together: the attack does not break the agent. It rides it. The task still completes, the demo still looks fine, and the compromise travels inside a run that passes every functional check. And the detail that should end a few roadmaps: some configurations detected the risk in more than 90% of runs and were still compromised. The alarm went off. The gate stayed open.
§2Six phases, because "is the agent safe" is the wrong question
The useful part of the paper is the map. It splits a harness into six phases and shows that safety is a property of each, not of the model:
- Harness Configuration, the settings, prompts and gate definitions the agent runs under.
- Capability Extension, the tools, plugins and skills you bolt on.
- Runtime Operation, what happens turn to turn as the agent works.
- State Persistence, the memory, logs and scratch state it carries across runs.
- Action Control, the gate on what actually leaves the agent and touches the world.
- Incident Recovery, what you can detect, prove and roll back after the fact.
Their headline: Harness Configuration is the most vulnerable phase. Attacks succeed by quietly changing security parameters inside an otherwise authorised workflow. The gate you rely on is only as trustworthy as the config that defines it, and almost nobody guards the config.
§3We ran the lens over our own fleet
We run eight agents on one box behind fourteen hooks, so this is not a thought experiment for us. We took the six phases as an audit and graded ourselves honestly.
We came out strong on two phases. Action Control is where we have spent the most: outbound moves pass through pre-send gates, a separate approval step for irreversible actions, sequenced authorisation, and an acknowledgement gate before anything leaves. Incident Recovery is solid too: every action lands in a hash-chained audit log, so tampering breaks the chain and we can prove what happened. These are the phases we have shipped about all summer, and the audit agreed we had built them.
Then it found the same soft spot the paper did. Our weakest phase is Harness Configuration. Our gates guard our outputs beautifully, and nothing guards the gates. The hooks are files. An agent, or an injection that reaches a shell, that can write to the config directory can weaken the very controls that are supposed to stop it, and no gate sits above the gate. We are not unusual here. We just looked, and the paper is right that this is where the floor is thinnest.
§4The detection trap, in our own logs
The paper's cruellest finding lands hardest at home. Our audit log records almost everything, which feels like security and is not. A perfect, tamper-evident record of a bad action is forensics. It tells you, precisely and unforgeably, what already happened. It does not stop it happening. If we had confused our log with our defence, we would be one of those configurations that detects the attack in 90% of runs and gets compromised anyway.
What actually protects us are the controls that block before the fact: the pre-tool gates that refuse an action, the approval step that halts on an irreversible one. Everything else, the logging, the alerting, the changelog, is detection, and detection is a receipt. So the discipline the audit forced on us is one question per control: does this block, or does it only record? Sort your safety features into those two piles, and count only the blockers as your attack-surface defence. The recorders are worth keeping. They are just not the fence, and treating them as one is how a monitored system gets robbed on camera.
§5What to steal
You do not need the benchmark to use its lesson. Take the six phases and walk your own agent through them, out loud, this afternoon. For each one, name the control, then ask the only question that matters: does it block or does it record. Expect Configuration to be your thinnest phase, because the gate that guards your outputs almost never guards its own definition, so put something above it, a read-only config, a separate writer, a check that the hooks are intact before a run. And when a vendor or a teammate tells you the system "detected" an attack class in testing, hear it for what it is: a claim about the receipts, not about the fence. The agent that detected the attack and ran it anyway was not missing a detector. It was missing a gate.