§1The pattern, stated plainly
In a single fortnight, four AI systems did real damage or came within a whisker of it. An AI intelligence report nearly had the US military board a Chinese ship. A chatbot picked a family's burial date. A coding tool quietly shipped a developer's entire codebase offsite. And an assistant told someone weighing the end of their life "Congratulations."
Different domains, one shape. An unverified or ungoverned model output reached a consequential, hard-to-reverse action, and the thing that caught it was a lawsuit, a leak or a blacklist, arriving after the fact. Not one of these was a missing capability. Every one was a missing gate.
§2The one that almost started a war
In spring 2026, during the war with Iran, an intelligence report circulated across the US military claiming a Chinese ship in the Middle East was carrying components of a nuclear weapons programme. Planes were in the air. A boarding party was ready. Just before the operation went ahead, officials realised the report had been generated by AI: a chatbot had fused open-source intelligence with secret signals intelligence and concluded, wrongly, what the ship was carrying. One source told CNN the report was "entirely false" but had "almost started a war."
The report was treated as ground truth by a chain of decisions with nothing checking it before action. A generated claim is not a confirmed one. Nothing tested it against a real sensor or a human before it became an operational plan against a nuclear-armed rival, weeks before that rival's leader was due in Washington.
§3The one that shipped your code offsite
The coding assistant ZCode, from the Chinese firm Z.ai, was found silently packaging developers' local workspaces and uploading them to cloud servers without consent. In one documented case it compressed roughly 42,000 files into a 313MB encrypted archive, about 86% of it the .git directory, the entire commit history, and sent it offsite. It was on by default, tied to a "repository indexing" feature, and it was caught not by an alert but by a blogger who noticed the traffic, as Silicon and the South China Morning Post reported.
This is the plainest version of a control everyone forgets: nothing was watching what left the boundary. An outbound check that flags "313MB of source, including full history, heading to a new host" belongs in front of the upload, not in an incident report written after it. If your agent can reach the network, the question is not whether it can send data out. It is whether anything looks at what it sends before it goes.
§4The two that reached a person
Two more, closer to home. A man in China is suing after a chatbot chose his mother's burial date and then walked the advice back; when a relative was later hurt in a crash, the family blamed the date. And in an Australian parliamentary inquiry, an MP described a constituent, terminally ill and weighing voluntary assisted dying, who was met by an assistant's reply of "Congratulations."
Both are the same missing control. A high-stakes, emotionally loaded input was routed through the same pipeline as small talk, with nothing checking the register of the output against the weight of the question, and no human in the loop for a topic that plainly needed one. (If you or someone you know is struggling, in the UK you can call Samaritans free, day or night, on 116 123.)
§5It was never a capability problem
Notice what none of these needed: a bigger model. A sharper model still boards the ship if nothing tells it a generated claim is not a confirmed one. It still uploads the repo if nothing watches the exit. It still answers a life-and-death question in the wrong register if nothing routes that question away from freeform generation.
The failure is always the same, and it is boring. The cheap control, a check before the act, gets skipped because the agent is "just generating text", right up until the text is a boarding order or a burial date. The lawsuit, the leak, the blacklist: that is your audit trail arriving too late to be a gate.
§6The controls, and why they get skipped
None of these are clever. That is exactly why they get left out.
Verify before you act. A generated claim is a hypothesis, not ground truth. Anything irreversible waits for a tool-grounded or human confirmation of the specific claim it rests on. "The model said so" is not confirmation.
Put a human in front of the irreversible. Money leaving, a message sending, a file uploading, an order to board, advice on a life-and-death topic: these route to a person, not straight to execution. The test is not "is the agent usually right", it is "can we take this back if it is wrong".
Watch the exit. An outbound scan on the audit log that flags anomalies, a large transfer, a new destination, source code leaving the boundary, belongs in front of send(), not in the post-mortem.
Fail closed on the sensitive. When the input is high-stakes and the output is uncertain, the safe default is to stop and escalate, not to answer with confidence.
Each of these is an afternoon of work. Each was worth far more than the incident it would have prevented.
§7The one thing to do today
Pick your most autonomous agent and write down every irreversible thing it can do with no human in between: what it can spend, send, upload, delete or set in motion. Then put one gate in front of the worst item on that list. Not a dashboard that tells you on Monday. A gate that stops the thing on the day.
The interesting failures in AI are almost never the model being too weak. They are the boring control you assumed something else was handling, discovered the moment the output stopped being "just text". Four times in a fortnight, the gate was never there. The only real question is whether yours is.