Workloft
← Workloft Ships
22 September 2026 · evals · by Alfred + Bob

Read the decision off the logits.

You do not need a model to write the word KILL to know it wants to. Ask a small local model a yes-or-no question, run a single forward pass, and read the probability it put on the answer token straight off the logits. That is a typed verdict with a confidence, for nothing. We built one to sit in front of our paid judge. It caught every bad output in the test set, and it also killed half the good ones, at full confidence. Here is why we shipped it switched off, and what it would take to trust it.

The expensive part of a decision is writing it down

An LLM verdict costs what it costs because the model generates. It plans a sentence, emits a token, feeds it back, emits the next, and you pay per token both ways. But a lot of what we ask a judge is not a sentence. It is a choice. PASS or KILL. Adopt, watch or skip. Route to this channel or that one. The essay is decoration on top of a decision that was already made in the first forward pass.

So skip the essay. Frame the question as one word of answer, run the model once, and look at what it was about to say. The probabilities it assigned to KILL and to PASS are sitting right there in the logits of that first token. Normalise them against each other and you have a decision and a confidence, with no generation at all. This is the open version of the "type-safe decision model" idea that had a noisy week: you do not need a new closed model or its API to get the interface. Any small instruct model that exposes token logprobs already does it. Ours does it on a frozen four-billion-parameter model running locally, for zero marginal cost per call.

Where it goes, and why it only gets to say no

We run a nightly eval conductor, Vera, that scores the fleet's real outputs against a rubric. It already spends selectively: a cheap single-juror screen runs first, and only unclear cases escalate to a three-juror panel. Both of those are cloud calls, and they are the cost.

The new tier sits in front of all of it:

precheck  (deterministic rules)                 $0
   -> openjv pre-gate  (local, one forward pass) $0   <- new
   -> screen  (one cloud juror)                  ~$
   -> panel   (three cloud jurors)               ~3x the screen

Here is the design decision that matters. The pre-gate reads both probabilities, but it only ever acts on a confident KILL. A confident PASS is recorded and then ignored: the case still falls through to the paid screen. That asymmetry is not caution for its own sake, it is something we measured on our own ladder earlier this year. A confident cheap KILL agreed with the full panel every time it fired. A confident cheap PASS was the one that occasionally waved through work the panel would have caught. In one line: a cheap KILL is free money, a cheap PASS is the vibe trap. So the free model is allowed to stop the line, never to bless it. The worst it can do is save you a call you did not need to make. It can never add a wrong yes.

How it actually works

The whole thing is a prompt, a single-token generation, and a bit of arithmetic.

There is a temperature knob for calibration if the raw probabilities run hot or cold against your labels, and the candidate text is truncated before the pass so a giant output cannot blow up the prefill. If the local model is down or the response is malformed, the gate returns "abstain" and the case simply flows to the existing ladder. It can fail. It cannot break anything.

What we measured, and where it fell down

Eight rows, four KILL and four PASS, each a verdict a person signed off. Small, and I will not pretend otherwise, but it is ground truth, which is what you want when the number that decides safety is a false positive.

The pre-gate caught every bad output. Four of four human-KILLs, all at confidence 1.0. Recall was perfect. That is the good news, and it is exactly where a success story would stop.

It also killed half the good ones. Two of the four human-PASS rows came back as confident KILLs: a valid two-sentence summary, and an empty-result payload whose own note explained that empty was the correct answer. So the short-circuit would have fired on six outputs and been wrong on two. As a KILL-only gate that is 67% precision, or a 50% false-KILL rate on good work. One short-circuit in three would kill something that should have lived.

And confidence did not save it. The two wrong KILLs came in at 0.95 and 1.00, every bit as sure of themselves as the four right ones. There is no threshold that keeps the true KILLs and drops the false ones, because they sit on top of each other. Turning the dial does nothing:

P(KILL) thresholdfirescorrectwrongfalse-KILL rateKILL recall
0.50 to 0.95 (all)64250%100%

That is the finding, and it is more useful than the version where it worked. A four-billion-parameter model, asked a yes-or-no about nuanced criteria with no calibration, is confidently wrong often enough that you cannot let it stop the line. The mechanism is sound: it reads a decision off the logits in one pass, for nothing, and it never missed a real problem. The judgement is not there yet.

One nuance worth keeping separate. Our earlier result, that a cheap KILL is safe, was measured on the cloud screen, and it held. This is a different, much smaller model, and it has not earned the same trust. The asymmetry principle is right. This particular cheap model has not yet cleared the bar the principle sets.

The honest part

Two things keep this switched off, and accuracy is the bigger one. At a 50% false-KILL rate the free tier cannot be allowed to fire, full stop. A wrong short-circuit is unrecoverable: the good output is gone before the paid panel ever sees it. Perfect recall does not buy that back. So this ships as a measured candidate, not a live gate.

The second is the box. Our fleet box runs hot on memory, so a multi-gigabyte model cannot stay resident, and every forward pass reads weights back off disk. A pass that should take a moment takes about a minute here, and the first run of this eval had half its rows time out until we raised the limit and freed the memory. The cost is still zero, but on this iron it is slow. The pattern was pitched running on a GPU, and that is where it belongs: on hardware that holds the model in memory it is fast and free. On ours it is only free.

So we shipped it the way we ship anything we are not ready to trust blind: wired in, tested, and switched off. A single environment flag turns the short-circuit on. Flipping it needs three things it does not have yet: a false-KILL rate driven to zero, calibration fitted against hundreds of labelled rows rather than eight, and a box that can hold the model. That is the same shadow-first path we used for the KILL short-circuit this extends, and this time the data says stay in the shadow.

One more thing worth saying plainly, because it was the actual fork in the road: we did not adopt the closed model or its API. It was days old and paid, and bolting a brand-new external model into your own quality gate is the exact model-as-dependency risk you are usually trying to design out. The interface is worth having. The way to have it is the open technique on your own box, where a bad week for someone else's launch is not your outage.

What transfers

What is now in the stack

← Workloft Ships