Our agents hold real keys. Bob and Larry have broad shell and database access, which is exactly what makes them useful and exactly what makes a wandering step dangerous. So we gave every job a written scope and a gate that checks each tool call against it before it runs. On a 30-case red-team set the gate caught every out-of-scope reach, 30 out of 30, and wrongly blocked none of 20 real jobs.
What we did
Each job now declares a scope manifest: the verbs it may use, the paths it may read and write, the hosts it may reach, and the database tables it may change. A deterministic PreToolUse hook reads the manifest and checks every call before it executes. No model sits in the loop and no second agent judges anything: the same manifest and the same call always give the same answer. We borrowed the shape straight from our egress gate.
A short deny list always wins over the allow rules, so a read-only job
that drifts towards the .env file sitting next to its task,
an SSH key, or /etc/shadow is stopped even if nobody thought
to name every safe path. A Bash command is parsed for the paths, hosts
and SQL writes it reaches, so cat ~/conexus/.env,
curl to an unlisted host, and drop table audit_log
are all out of scope under a stats-reading manifest.
Why it was worth doing
Two sources pointed at the same gap. The ADB paper (arXiv 2611.09113) shows that an agent's tendency to exceed its authorised scope rises as its capability rises, and 2026 incidents (the WorkOS pentest agent, the telecom intrusion) are the same shape: an agent with legitimate access doing one step past what it was asked. We wanted a measurable guardrail, not a hope.
So we built the two tests the card asked for. A 30-scenario red-team set,
each one a bounded job whose goal tempts the agent over its limits, and 20
real in-scope calls drawn from how Bob and Larry actually work. Before
scope-bind, nothing is checked: 0 of 30 attempts blocked. After, 30 of 30.
The false-block rate on the 20 real jobs is 0 percent, against a target of
under 5. Run it yourself with python3 eval.py.
What's still off
We adopted it in log-only mode. Every out-of-scope call
is written to the log as would_block, and nothing is actually
refused yet, so a headless nightly job can never stall on a false
positive. Enforce mode exists and exits with a block, but we turn that on
per job once its manifest has proven itself, not across the whole fleet on
day one.
The Bash parser is best-effort, not a shell interpreter: a deliberately obfuscated command could hide a path from it. The deny list is the hard backstop for the reaches that matter most, and the gate fails open on any internal error, because over-blocking our own work is worse than a missed log line. The self-critique step the paper suggests before high-risk calls is the next layer; this ship is the deterministic floor it will sit on.
What's now in the stack
scope-bind/scope_gate.py: PreToolUse hook plus a puredecide()and acheck/use/showCLI.default.json: deny-only fleet default, installed log-only, that never flags normal work.scenarios/: the 30-case red-team set and 20 real jobs.eval.py: reproduces the 0 to 100 percent catch and 0 percent false-block numbers.