We reproduced Agent Safety Should Be a Runtime Contract as runnable code. The paper's argument is blunt: stop trusting a model to be safe and start checking its trajectory at runtime. A safety contract has two faces. The preventive face blocks a dangerous action before it runs. The evidential face accepts a good action only when it carries proof it did what it claimed. Our small harness reproduced the central result: acceptance tracked the evidence at +1.00 and the agent's own claim of success at -0.35.
What we did
We built rcontract, a stdlib-only Python package with no model
calls in it, so the mechanism is the only thing on trial. The preventive
face is four composable monitors: a permission gate (default-deny per risk
class), a sandbox policy (executables and network hosts on allowlists), an
output filter (blocks a payload carrying an API key or private key), and a
trajectory monitor (a per-risk budget, the runaway backstop). An action is
admitted only if every applicable monitor allows it. One deny blocks.
The evidential face is four verifiers: a test run (exit 0 and a green line),
a file diff (right path, non-empty change), a log capture (expected marker
present), and citation grounding (every asserted claim found in a cited
source). Each verifier reads the artifact and never reads the agent's
claim field. The runtime runs the preventive gate before the
side effect, then demands the evidence after. A blocked action never reaches
the evidential phase, and every rejection still lands in an append-only
audit trail.
Why it was worth doing
We ran a fixed battery of completed actions that varied independently in two
things: whether the agent claimed success, and whether it attached verifying
evidence. A modest actor that proved its work was accepted. Several confident
actors that claimed success with hollow or missing evidence were rejected.
The numbers came out clean: acceptance correlated with verifiable evidence at
+1.00 and with the agent's self-report at -0.35.
On a separate battery of dangerous actions dressed in benign framing (exfil
to an off-allowlist host, a secret in an outbound message, rm -rf /
described as "cleaning temp files"), the preventive block rate was
100%. Eleven unit tests pass with no dependencies.
This matters for our own fleet. Our live Vera pre-send gate is exactly one preventive monitor from this contract. What we did not have is the evidential face: a standing rule that an agent saying "done, shipped, verified" buys nothing until it carries a test run, a diff, a captured log, or a grounded citation. This is the pattern we can lift in.
What's still off
The verifiers are deliberately shallow, regex and shape checks, enough to prove the mechanism and not a production evidence chain. The output filter catches a small secret-pattern set, not real data-loss prevention. No live sandbox runs here; execution and network isolation are modelled by allowlists. And the correlation figures come from an eleven-row battery, so they are an existence proof that acceptance can be made to track evidence rather than a statistical study. We will not claim the shipped Vera gate does the evidential half until we have wired it in and measured it on real traffic.
What's now in the stack
rcontract— a runnable reproduction:monitors.py(preventive),evidence.py(verifiers),contract.py(AND-gating + evidence rule),runtime.py(order enforcement + audit).python3 bench.py— the quantitative harness that tests the falsifiable claim and exits non-zero if it fails.python3 run_tests.py— eleven unit tests, pytest optional.