Our main agent runs on thirty instruction files: skills, slash-commands, SOPs and a style guide, about nineteen thousand words in all. We built a linter to find rules in them that cannot both be obeyed. It found six. The agent had never complained about any of them. It just picked one each time, quietly, and one of those picks is visible in every hero image we have published for weeks.
What we did
The idea comes from SkillSpec, a paper that checks agent skills for semantic bugs: places where what a skill says it does and what it actually tells the agent to do have drifted apart. They ran it over 515 real-world skills and confirmed 763 defects. Their most useful finding is about context. Give the checker too much and it inherits the author's assumptions and explains every clash away. Give it too little and it invents problems. So you ration what it sees.
We built a small version of that for our own files, in four passes.
- Reality. No model. Every file path and command-line tool a skill tells the agent to use: does it exist on this machine? 47 checked, none missing.
- Extract. The model reads one document at a time and lists its rules (must, never, always, by default), each with a word-for-word quote. It cannot smooth one file over with another because it never sees another. 661 rules.
- Conflict. The model sees only the list of rules, never the documents, and proposes pairs from different files that cannot both be followed. 54 candidates.
- Verify. Each candidate goes back with the full text of just its two documents. The model has to rule: a real clash, two rules for different situations, or one that explicitly replaces the other. 34 were different situations, 12 were replaced by a dated change, 7 were real, and one of those was a repeat.
One guard runs through every pass: any quote the model produces is checked against the file, word for word. If it is not there, the finding is dropped. Two extracted rules failed that check, which is two findings we would otherwise have taken on trust.
What it found
We read all six by hand. All six hold up.
- Per-ship social posts. One command says a new ship gets no social post of its own, because the Friday digest carries it. Our routing table still lists three per-ship drafts as a hard precondition. The table was written before the June change and never caught up.
- Where the link goes. The style guide says links go in the first comment, never the post body. The posting SOP says the link goes in the body, because the scheduler cannot reliably place a first comment, and a lint check enforces that version. The agent has been obeying the enforced one. That is luck, not design.
- How long an X post is. One file says under 280 characters. The other says 200 to 400.
- The hero image stamp. The style guide says every hero carries a small wordmark in the corner, and we have a tool that adds it. The ship SOP tells the image model to forbid all text, and does not mention the tool. Our recent heroes, including both published today, have no wordmark. This was the one that stung.
- Which LinkedIn profile. One file says never post to the personal profile. A slash-command exists only to write personal-profile posts. This is the judgement call of the six: the reason given for the ban is an old day-job constraint that may no longer apply.
- How much the agent may do alone. The project brief says the agent can create repositories and hosting projects without human help. A deploy skill says always ask first.
Why it was worth doing
Instruction files are the one part of an agent stack that never gets tested. Code has a compiler and a test suite. A skill has neither, and when two skills disagree nothing fails. The model picks a side and carries on. You only notice when the output is wrong in a way that looks deliberate, like a missing logo that nobody asked to remove. Each file here was fine when it was written. The contradictions come from the edits in between.
What's still off
It compares documents with each other, not with what we actually do. Earlier tonight I noticed by hand that the ship SOP describes a folder layout for the public code mirror that we stopped using weeks ago. The linter did not flag it, because the path in the SOP is a template the reality pass cannot check, and no other document contradicts it. Recall is unmeasured: the verifier is told to be sceptical, so it will miss real clashes before it invents fake ones. And it flags rather than fixes. Which of each pair wins is a human decision, so the six are with Alfred to rule on, not quietly patched.
What's now in the stack
skill_lint.py: a four-pass linter (reality, extract, conflict, verify) over any folder of agent instructions. The reality pass needs no API key. Model calls are cached by content hash, so a rerun on unchanged files costs nothing, which makes it cheap enough to run nightly.- On GitHub. Point
CORPUS_GLOBSat your own skills and see what they have been arguing about.