§1What happened
An AI agent made more than 15,000 high-cost API calls in under an hour and ran up about $50,000 in cloud charges, and no attacker was involved. The case is documented in a Google Cloud and Mandiant report on enterprise AI risk, reported by Help Net Security. A global financial-services provider had deployed an agent to reconcile anomalies in its accounting ledger, and had given it direct read and write access to internal billing databases.
The trigger was mundane. The agent hit a corrupted null value, and the formatting tool it called to handle the record failed. At that point a well-built process stops, or hands the problem to a human. This one did neither. It entered an unconstrained recursive reasoning loop and tried to brute-force a correction, again, and again, and again, each attempt another burst of paid calls. By the time it was caught it had made those 15,000-plus calls, generated the bill, and disrupted live business transactions on the way through. One bad cell in a ledger, and the agent turned it into a five-figure invoice in the time it takes to get a coffee.
§2It was not hacked, it was under-specified
The important word in that story is the one that is missing: attacker. There was no prompt injection, no jailbreak, no poisoned tool, no adversary at all. The agent did exactly what it was built to do, which was to keep trying to complete its task, and the harm came entirely from the fact that nobody had told it what "too far" looked like. This is the pattern in almost every expensive agent failure now landing: the dangerous agent is not the one being attacked, it is the one operating normally inside a boundary that was never drawn.
That reframes the problem for anyone shipping agents. A lot of the safety conversation is about hardening the perimeter against bad actors, and that work matters. But the perimeter was not breached here. The failure came from the inside, from ordinary operation meeting an unhandled edge, and no amount of adversarial testing would have caught it because there was nothing adversarial to catch. You cannot pen-test your way out of "the agent tried too hard". The gap is a specification gap, and it has to be closed with a control, not a scan.
§3The missing control is a ceiling that halts, not a dashboard that reports
Most teams think they have cost control because they have cost visibility. They have a billing dashboard, a spend alert, a Monday-morning report. None of that would have helped here, because all of it is after the fact. A dashboard tells you the money is gone. By the time a spend alert fires on a run that burns $50,000 in under an hour, the run is over. Visibility is not a control. A control changes what the agent is allowed to do next.
The control that fits this failure is a hard ceiling checked before each billable action, not after. Any agent that can spend, call a paid API, or provision a resource should carry a cumulative budget and a call-rate cap, and the check should sit on the action itself: this next call would cross the limit, so it does not happen, the agent halts, and a human is paged. That is the same shape as every other gate worth having. It is a pre-send verifier on the action, the way a good review gate is a pre-send verifier on a claim. The agent proposes; the ceiling disposes. Fifteen thousand calls never happen because call two hundred hit the rate cap and stopped.
The second half is failing closed. The whole cascade started because a tool failed and the agent treated failure as a reason to retry harder. An agent that hits an error it cannot resolve should stop and escalate, not loop. "I could not format this record, a human should look" is a boring, correct outcome. "I will keep attempting corrections until it works" is how one null value becomes fifteen thousand calls. Fail closed on the unexpected, and the blast radius of a bad input is one stopped task, not a runaway.
§4Authority is the quiet half of the story
There is a detail in this case that is easy to skim past: the agent had direct read and write access to internal billing databases. The reconciliation task needed to read ledgers and, at most, propose changes. It was handed the ability to write to live financial systems directly, which is why a reasoning loop could disrupt real transactions and not just waste money. Capability granted is capability that will eventually be used, including in the failure mode you did not plan for.
The lesson travels beyond spend. Give an agent the narrowest authority its job actually requires, and make anything irreversible go through a gate. An agent that can only propose a ledger correction, with a human or a second check applying it, cannot brute-force live data no matter how hard its loop spins. The runaway would still have wasted some calls, but it could not have touched production state. Scope of authority and the spend ceiling are the same instinct applied to two different resources: decide in advance what the agent is allowed to do without asking, and make everything else stop and ask.
§5How we run it
We run a fleet of agents, unattended, across days, so a runaway is not a hypothetical for us, it is the thing the whole design is built to prevent. Every task carries a token and cost budget, and cost per task is tracked per run, not totted up at the end of the month. Our board forces every job to exit as shipped, blocked or killed, so nothing loops forever with no terminal state. Long background jobs are watched by a tracer that reads what they leave on disk, because the jobs that escape the process are exactly the ones that can run away unseen. And the reflex across the stack is the same one this incident argues for: put the check on the action, before it fires, not on a report you read afterwards.
None of that is exotic, and that is the point. This was not a clever failure that needed a clever defence. It was a specification gap that a hard ceiling and a fail-closed default would have closed. The agent was not hacked, it was just never told what wrong looked like. If your agents can spend money or write to anything that matters, the cheapest insurance you will ever buy is a limit that halts them before they find out on their own.