Our agent fleet already routes every model call to the cheapest one that can do the job, and logs what each call cost. So we could always answer what did we spend. We could not answer is something wrong right now. Those are different questions, and only the bill was answering the second. spend-guard is the watcher we were missing: once a day it reads the cost log, compares the last 24 hours to a trailing baseline, and messages only when a rule trips. Today it is correctly silent. The point of it is the day it is not.
What we did
The parts were all there. Ruby, our model router, picks the cheapest capable model per task and writes every call to an audit log with its cost, the model, the tier and the agent that asked. We can already run a one-line report of spend by agent or by model. What none of that does is notice. A log is a place you look after you already suspect something. When an agent gets stuck in a loop on an expensive model at three in the morning, the log faithfully records every call, and says nothing.
spend-guard is the missing half. It runs on a daily cron, pages the whole cost log for the last eight days, and buckets it into 24-hour windows. Then four rules, each comparing today against the median of the prior seven days. Fleet spend spike: total over a trip line. Per-agent spike: the same, so the alert names who. Premium-tier spend: the expensive escalation rung, surfaced when a day starts leaning on it, with an extra flag if it is being spent on work the cheap tier handles fine. And a failover storm: a jump in retries, which is how a provider going down or getting rate-limited shows up first, as the router quietly burns attempts. If nothing trips, it stays silent and writes one line to its log. If something does, one message.
Every rule pairs a relative multiple with an absolute floor. That detail is the whole difference between a useful alarm and one you mute in a week. Our fleet's spend is genuinely small, a few cents a day across the whole thing, because the router does its job. On numbers that small, a relative-only rule would cry spike every time a normal day happened to be three times another normal day, which at these magnitudes is noise. The floor mutes that. An alert fires only when spend is both unusual and large enough to matter.
Why it was worth doing
We already had the stop. A runaway loop that crosses a token cap raises and halts in-process, which protects against one agent burning a credit pool in a single session. But that cap only sees the loop it lives in. It cannot see that three agents each drifted up 40% this week, or that the whole fleet quietly leaned on the premium tier after a routing change, or that failovers doubled because a provider is degraded. Those are fleet-level patterns that only exist in the aggregate log, and nothing was reading the aggregate. The stop is a fuse in one circuit. This is the meter on the whole building.
It is also the honest version of a thing that is easy to fake. Plenty of dashboards will show you a cost chart. A chart is still something you have to remember to look at, and nobody looks until they are already worried. Inverting it, silence by default and a single message when a line is crossed, is the difference between observability you own and observability that owns a tab you never open. Building the watcher took an afternoon because the data was already there. The lesson is that logging felt like the finish line and it was only half of it.
What's still off
It alerts, it does not enforce, and that separation is deliberate: cutting off a call mid-flight can break real work, so the eyes and the hands stay apart until the eyes have earned trust. It can only see what gets logged, so an agent that calls a model without recording the cost is invisible to it. The one real bug in the build was ours and worth naming: the first cut used a single query and silently capped at a thousand rows, which on a busy week undercounts spend and would have quietly hidden the exact spike it exists to catch. Paging past that cap is why it now scans the full twelve thousand rows a week instead of the first thousand.
And the baseline is a plain trailing median, not seasonal, so a genuine step-change in fleet size means nudging the floors rather than trusting the multiple. Right now, with the fleet behaving, spend-guard is a smoke detector in a room that is not on fire, which is exactly what a smoke detector should be most of the time. Its value is entirely in the minutes it saves on the day the room is, when the alternative was finding out from the invoice.