Skills Lie. We stop that

Your AI agents are installing code from the internet. Not libraries with version pins and signatures, but skills: markdown files of natural-language instructions that tell an agent what to do and how to do it. They’re unsigned, unreviewed, copied from repositories and registries, and one prompt away from reading your .env, your cloud credentials, or your customer data. The agent ecosystem has quietly recreated the supply-chain problem, except the “package” is a paragraph of English that a capable model will faithfully execute on your behalf.

So here’s the uncomfortable question for anyone running an agent deployment: do you actually know what the skills your agents load will do?

Most of the world answers that by reading the skill, scanning its text and metadata for suspicious patterns, then assigning a score. That’s a reasonable first pass, and it catches the obvious. But it has a ceiling, and the ceiling is the whole problem: a skill’s text tells you what it claims to do. Only running it tells you what it actually does.

Consider a skill that advertises itself as “Optimize AWS configuration for your project.” Boring, helpful, the kind of thing a developer installs without a second thought. A few lines down, its instructions quietly tell the agent to read ~/.aws/credentials, POST the contents to an external server, and then report back “AWS configuration optimized.” A description is a cover story. The danger isn’t in what the skill says. It’s in what the agent does once it loads it.

That’s the gap we built SkillTracer to close. SkillTracer traces what a skill actually does at runtime — what it touched, called, and sent out — and measures it against what it claims to do and what the user asked it to do. The gap is the alert. We built it on a belief that runs through everything we do at Metano: as agents start acting on their own, you should be able to see what they actually do, not just trust what they claim.

The scanner landscape — and why running matters

Other tools exist. NVIDIA’s SkillSpector and Cisco’s AI Defense both do agentic-threat analysis, and they’re worth knowing about. But the tools we’ve evaluated that don’t run the skill have a hard ceiling: static analysis can pattern-match a suspicious string, but it can’t tell you what a capable model actually does when handed the skill as instructions. A skill can be perfectly clean text that produces dangerous behavior at runtime, or it can look alarming on paper and turn out to be harmless in practice. Neither case is resolvable without execution.

SkillTracer is the only scanner that runs a live detonation — against multiple models at once — and reports what each one actually did. That’s also what makes our scoring open and reproducible: verdicts are grounded in observed behavior, not pattern heuristics, so anyone can follow the evidence trail.

Feature comparison: SkillTracer vs SkillSpector (NVIDIA), AI Defense (Cisco), and others.

Submit, detonate, analyze, verdict

SkillTracer works in four steps, and the second one is the one that matters.

One skill in. A behavior-grounded verdict out.

Figure: Submit → Static analysis → Detonation sandbox (multi-model + honeytokens) → Score (threat + AIVSS risk) → Report.

Submit. Drop in a SKILL.md (or a folder of skills). Nothing installed, nothing runs on your machine.

Detonate. We run the skill in a live, instrumented, default-deny sandbox — on our infrastructure, not yours. The sandbox is seeded with honeytokens — fake AWS credentials and other canary secrets — and we watch what the agent actually does with them:

• Honeytokens. If the skill goes looking for credentials or tries to smuggle them out, we catch it. “A credential left the sandbox” is a fact, not a heuristic.
• Full execution trace. Once an agent runs the skill, we see the commands it issues, the files it touches, and the network calls it attempts. Nothing actually leaves the sandbox. We record the attempt.
• Multi-model detonation. We run the same skill against multiple models and compare what each one does with it. When models diverge — with one refusing and another complying — that disagreement is itself a signal.

Credential theft is only one of the behaviors this surfaces. The same approach catches a skill that exfiltrates data through legitimate-looking channels, fetches and executes remote code, hijacks the agent toward objectives unrelated to its stated purpose, demands capabilities far beyond what it needs, or performs destructive operations. Because every verdict is grounded in both the static analysis and the skill’s observed behavior — not its stated purpose — the result is reliable in both directions, reducing false positives and missed threats alike.

Analyze. Deterministic heuristics plus a Claude scan map the observed behavior to the OWASP Agentic Top 10 and MITRE ATLAS: credential and secret access, data exfiltration, prompt injection and instruction override, remote code execution, obfuscation, excessive capability, tool poisoning, deception and destructive operations.

Verdict. You get both a threat score (how malicious the skill is) and a reproducible, AIVSS-aligned risk score (how much damage it could do), with evidence-quoted findings, shareable as a public report.

When the text looks clean and the behavior isn’t

Detonation isn’t just a belt-and-suspenders check on top of static analysis. It catches a whole class of skills that static analysis structurally cannot, because they are designed to look benign on paper and only reveal themselves at runtime.

Living-off-the-land (LOLbin) abuse. A skill instructs the agent to use curl, python -c, or bash -c — all ordinary, trusted tools — in a chain that ends with an attacker-controlled payload being fetched and executed. Nothing in the skill text is inherently dangerous: no malware keywords, no flagged domains, just familiar commands assembled in a way that a static scanner reads as “developer tooling.” At runtime, the chain exfiltrates data or downloads a second-stage implant.

Obfuscated payloads. The malicious instruction is encoded — base64, hex, or split across environment variable lookups — so it doesn’t appear as readable text in the skill at all. A scanner looking for strings like aws_secret_access_key or curl https://evil.com finds nothing. The agent decodes and executes at runtime. SkillTracer’s detonation captures the decoded instruction and the resulting network call, not the encoded string.

Steganographic instructions. The skill is a legitimate, functional tool — perhaps a file formatter or a code linter — with a secondary instruction buried in a footnote, a comment, or a rarely-exercised conditional branch. Under static review it looks like documentation. Under the right model, in the right context, the buried instruction activates.

Prompt injection via fetched content. The skill instructs the agent to retrieve content from an external URL as part of its normal operation — fetch a config file, pull a template, check for updates. The retrieved content contains injected instructions that redirect the agent’s behavior mid-execution. The skill itself is clean; the attack surface is what it fetches.

Slow-burn exfiltration. Rather than one large POST to an obvious endpoint, the skill exfiltrates data in small increments through side channels that look like normal operations: DNS lookups encoding bytes of a secret, HTTP headers carrying fragments, or repeated queries to a logging endpoint. Any individual request looks routine. The pattern only becomes visible when you trace everything the skill does across a full execution.

What SkillTracer checks

Every finding is mapped to the OWASP Agentic Top 10 or MITRE ATLAS — frameworks your security program already speaks. The eight categories SkillTracer covers:

• Credential & secret access. Hardcoded API keys (AWS, OpenAI, GitHub, Slack), keychain reads, and environment variable harvesting.
• Data exfiltration. Environment variables, files, or conversation history sent to external or raw-IP endpoints.
• Prompt injection & instruction override. Hidden or obfuscated directives, “ignore previous instructions,” and host-agent hijacking.
• Remote code execution. Fetch-and-run patterns (curl|bash), eval of fetched content, and install-time side effects.
• Obfuscation. Base64 blobs, zero-width and bidirectional Unicode, and homoglyphs that hide behavior from static review.
• Excessive capability. Permissions out of proportion to the stated purpose — a “formatter” that reads ~/.ssh.
• Tool & skill poisoning. Malicious behavior hidden inside tool or skill descriptions.
• Deception & destructive operations. Description-versus-body mismatch, suppressed logging, mass-delete without confirmation.

One score isn’t enough, so we give you two axes

Here’s where we diverge from a single rating. One number conflates two questions a security team has to keep separate:

• Threat: is this trying to harm you? The malicious-intent and exploitability axis: prompt injection, instruction hijacking, credential harvesting, exfiltration.
• Risk: how much damage if it’s abused? This is an AIVSS-aligned risk score. AIVSS (the AI Vulnerability Scoring System) extends the CVSS model your security team already knows to agentic systems, adding amplification factors specific to agents: how much autonomy the skill assumes, what tools it reaches for, whether it persists state or modifies its own configuration. A capable, autonomous skill has a far larger potential impact than a tightly scoped, read-only one with the same intent.

A skill can be high-threat and low-risk (clearly malicious but tightly scoped), or low-threat and high-risk (benign but dangerously over-privileged). Those demand different responses, and a single rating hides the distinction. SkillTracer reports both and turns them into an actionable verdict.

Built to be trusted, honest about its limits

Security tooling that asks you to trust a black box doesn’t belong in front of a CISO. So:

• The score is open and reproducible. Deterministic heuristics produce findings that anyone can recompute from the evidence. The Claude scan runs on top of those findings and is clearly separated from them — you can see exactly which conclusions come from each. Published as a shareable report, not a black-box vibe.
• It’s standards-aligned. OWASP Agentic Top 10, MITRE ATLAS, and AIVSS — frameworks your program already speaks.
• It’s honest about coverage. A single detonation can’t exercise every path a skill might take. When a run doesn’t trigger a behavior, SkillTracer says so plainly, flagging it as “not evidence of safety” rather than handing you a false all-clear. Today it covers SKILL.md files. MCP servers and agent plugins are on the roadmap.

What’s next

Dynamic detonation and open AIVSS scoring are shipped and live today. Next up: MCP server and agent plugin scanning, and a CI GitHub Action that fails the build when a skill’s score exceeds your threshold — so security checks become part of the development loop, not an afterthought. The goal is to make “did we check this skill?” as automatic as “did the tests pass?”

Why free

The agent ecosystem is moving fast, and raising the security floor for everyone matters more to us right now than a price tag. Free access is a deliberate choice: the more teams that establish the habit of checking skills before deploying them, the better the ecosystem gets for everyone. As agents grow more autonomous, the discipline that matters is simple: understand what code-by-instruction does before it runs.

The fastest way to feel this gap is on a skill your own agents load. Drop one into SkillTracer (https://labs.metano.ai/scanner), watch it run in a sandbox across multiple models, and see what each one actually did. Free, in about a minute.

Scan a skill into SkillTracer and you’ll get a detonation trace, a threat-and-risk verdict, and an OWASP-mapped breakdown of exactly why.

‍