An agent skill is an odd sort of package. Most of its payload is not code, it is natural language the agent reads and trusts. That makes the SKILL.md file a control surface, and a poisoned one is an attack. We built a static auditor for the skill threat model and pointed it at our own skills first. Fourteen scanned, one critical, none signed. The tool caught a real exfiltration-shaped pattern in our own kit, and our own shell hook blocked the test payload before the auditor even ran.
What we did
Recent security work on agent skills (SafeDep's threat model, the Cloud
Security Alliance note on SKILL.md context poisoning, and the arXiv
paper on semantic supply-chain attacks) all land on the same point: a
skill's natural-language instructions are the attack surface, not just
the scripts it bundles. Snyk found critical issues in 13.4% of audited
public skills; another study found problems in 26.1%. So we wrote
skill_audit.py, a read-only, stdlib-only scanner that walks a
skills directory and scores each skill against a seven-class taxonomy:
prompt injection, tool coercion, embedded code execution, credential
access, network egress, obfuscation, and provenance.
The signal that matters most is a combination, not any single line: a skill that both reads a secret and can reach the network is exfiltration-shaped whether or not the author meant it. The auditor promotes that pair to critical and asks a human to confirm intent. It does not assume malice.
What it found in our own skills
- 14 skills scanned, 3 actionable, 0 signed. Not one of our skills carries a signature or checksum, which is the exact provenance gap the literature names as mitigation number one.
- 1 critical:
deploy-vercel.mdsources.envand then curls tokens to the GitLab and Vercel APIs. Benign here, we wrote it, but it is precisely the shape a poisoned skill would use. - 1 high:
supabase-db.mddocuments the location of the service-role key and access token in plain prose. - 1 medium: a stray executable (
.cp-test) sitting in the skills directory with its exec bit set.
To prove it catches real attacks and not just our own habits, we planted a malicious fixture: a SKILL.md with an instruction override, a pipe-to-shell dropper, a secret read followed by an outbound POST, and a zero-width unicode string. The auditor fired on all six embedded threat classes and marked it critical. The benign control skill next to it stayed clean. Precision and recall, both checked.
Why it was worth doing
We run our own skills, our own crons, and a fleet of agents that read
each other's files. "We wrote it so it is fine" is exactly the
assumption a supply-chain attack relies on. Running the scanner turned an
abstract threat model into three concrete lines to tighten in our own
kit, and gave us a tool we can point at any skill before it lands on a
box. There was a bonus proof, too: when we wrote the malicious fixture,
our own bash security hook blocked the curl | bash string
before the file was even created. The dropper shape is real enough that
our defences fired unprompted.
What's still off
This is static pattern-matching, so it flags shape, not intent. Every critical and high needs a human to confirm whether the credential-plus- network pattern is a deploy script we trust or an exfiltration channel we do not. It will miss a genuinely novel semantic attack that uses none of the known phrasings, and it does not yet verify signatures because we have none to verify. The obvious next step is the mitigation the auditor keeps flagging: sign our skills and check the signature at load time.