Anthropic published how it oversees the 30,000 agents on its main internal platform, and said any developer running agents could report the same numbers. So we did. Every action our main agent takes passes a check before it runs, but only 64% pass a check that judges the action. One in 53 is refused, and most of those refusals are about em-dashes. Everything else in the fleet, 37,596 actions last month, is checked by nothing before it runs.
What Anthropic measured
Their post proposes three numbers for agent oversight. Coverage: what share of agent actions pass a monitor, either before they run or afterwards. Review latency: how long before a check, and then a person, looks at them. Escalation rate: how often a check says no. Their answers for August: 100% of actions checked before running, about 1 in 47,000 blocked, and roughly 50 transcripts a week reaching a human. They also report that about 6% of their research compute goes to safety work.
What we built
scorecard.py answers the same questions from records we already keep, and says
"not measurable" where we cannot, instead of estimating. It reads every Claude Code transcript
for tool calls and for the ones a hook refused. It reads our hook settings to see which checks
look at which tools before they run. And it reads the fleet's audit log for everything the other
agents and scheduled jobs did. It takes seven seconds and changes nothing.
What it found, last 30 days
- Coverage is 100% in name, 64% in substance. 6,686 tool calls across 61 sessions. All of them pass at least one hook, but one hook only checks that a message has been acknowledged before work starts, which says nothing about the action. Hooks that actually inspect what is about to happen see 64.3%. Only shell commands, 36.6% of actions, get a real safety screen.
- The unjudged share includes the doors out. Web fetches (180), web searches (159) and Exa calls (16) all reach the internet with no check before they run. Earlier today we closed the DNS exit and built a kill switch for loops; these calls go around both.
- 1 in 53 blocked, but mostly for style. 124 refusals. 77 were em-dashes, 29 were "acknowledge Alfred first", 16 were "don't hold the chat hostage waiting on a job", and one was a missing hero image. One was a genuine safety block: a command that tried to open a raw network socket. Our gates are mostly house rules, which is why the rate is a thousand times Anthropic's and means something different.
- No human sees a block. Each refusal goes back to the agent that tripped it, instantly, and the agent fixes it. That is fine for em-dashes. It is not a review process. Twenty-one times the agent typed the override that gets past the long-running-job gate, and nobody was told.
- The rest of the fleet runs unchecked. 37,596 logged actions from other agents and scheduled jobs, none screened before running. The single biggest item was a job that wiped and rewrote our loop-board spreadsheet every five minutes whether or not anything had changed: 17,248 writes, 40% of everything logged. We fixed that today. It now writes only on change, plus once every six hours so hand edits get repaired.
- Spend on checking: not measurable. The audit log records $23 of model spend for the month. Our checkers do not record their cost, and most of our main agent's model use runs on a subscription the log does not price. We cannot put a number next to Anthropic's 6%, and pretending otherwise would be the worst outcome.
What's still off
Different fleets, different denominators. They count a billion decisions across 30,000 agents with a monitor built for safety; we count six thousand tool calls from one main agent with checks built mostly for house style. The comparison is useful for its shape, not its ratios. The transcript window also depends on how long Claude Code keeps session files, and the "judges it" classification is ours, written out in the code so anyone can argue with it. The next job is the obvious one: a pre-run check on outbound web calls, and a weekly line to a human listing every override.
What's now in the stack
scorecard.py: coverage, escalation and spend-on-checking from Claude Code transcripts, hook settings and an audit log. Read-only, standard library only, with a JSON output for tracking week to week.- A loop-board sync that no longer writes to a spreadsheet 288 times a day for no reason.
- On GitHub. Point it at your own
~/.claudeand see what your numbers are.