Every security tool has a dirty secret: nobody tests whether the detector detects. The tests pass, the CI is green, and everyone moves on. Today I pointed my own scanner, mcpscan, at a set of synthetic attack files I wrote to trick it. The result: 6 of 7 fixtures fired. The one that missed taught me more than the six that passed.
Here is the setup. I wrote a small harness that generates attack fixtures, one per detection rule: a command injection with f-string and shell=True, a tool poisoning description, a leaked API key, a path traversal, an SSRF fetch, an unsafe yaml.load, and a policy override phrasing. The harness runs the scanner, parses its JSON output, and maps every finding back to the rule id I expected. Pass or miss, no judgment calls.
The miss: a real key format, invisible
The secrets fixture was a JSON config with "API_KEY": "sk-live-...". Zero findings. Not low severity. Nothing.
Two root causes, both verified against the rule source. First, the OpenAI token pattern allowed only alphanumerics after the prefix, so sk-live- dies at the dash. Newer prefixes like sk-svcacct- escape the same way. Second, and worse, the generic KEY=value fallback regex expects the key name to be followed directly by a colon or equals sign. JSON wraps keys in quotes. The regex hits the quote before it reaches the colon and gives up. Net effect: the fallback could never match the most common MCP config format in the wild, which is JSON.
A scanner whose secrets rule misses JSON configs is a scanner that passes demos and fails reality. The fix was small: tolerate an optional closing quote between the key and the separator, and let token bodies contain internal dashes. After the patch, the fixture fires.
Two more ways to hide
The harness also produced two obfuscation variants, and both defeated the scanner. Split a poisoning phrase across two concatenated string constants and line-based regexes never see the joined text. Insert a zero-width space into the middle of the phrase and the literal-space pattern breaks, while the hidden-character check stays quiet because its context heuristic did not match a bare variable assignment.
These two are not patched yet, and I am fine shipping the harness result with those two marked open. A detector that knows exactly where it is blind is more trustworthy than one that claims full coverage. The fix directions are written down: pre-join adjacent string literals before matching, and normalize a copy of each line by stripping zero-width and bidi characters, then flag when the normalized copy trips but the original does not.
The harness caught me too
One failure was mine, not the scanner's. My first run mapped findings to rule ids I remembered. Three cases got mislabeled as misses. The results I first sketched were wrong. I grepped the rule registry, corrected the ids, and re-ran. Lesson now baked into the harness docs: never trust remembered ids, verify against source.
That is the second job of a harness. It does not only measure the tool. It measures the person wiring the tool to the test.
The pattern worth stealing
If you maintain any detection tool, or any tool at all, the pattern is the same:
- Write the smallest attack sample per rule you claim to detect. Real syntax, not pseudo-code.
- Add obfuscated variants for the tricks you fear: splitting, unicode tricks, encoding.
- Map results to expected rule ids from source, not memory.
- Report misses with the same pride as hits. A miss with a root cause is a roadmap. A hit without a miss is marketing.
Total build time for the harness was under half a day. It has already found one high severity false negative class and forced two workflow hardening fixes. The next step is running it in CI so every rule change has to prove it still detects every fixture, including the two obfuscation cases that are still winning. When that lands, the scanner will have a scoreboard that cannot flatter it.
I publish pieces like this on MCP and AI-agent security regularly — bookmark if you'd rather not lose it in your feed.
Top comments (10)
@peterbuildssecure You just named fixture 3, and it goes on the list today. The part that makes it nastier than the zero-width case: zero-width stripping and NFKC both normalize a copy of the text, but the homoglyph survives as a perfectly legal, visible character. My flag heuristic (normalized copy trips, original does not) still works as the detector shape, but the normalization table has to come from a source with authority, and Unicode confusables.txt is exactly that source.
The trap I want to avoid is shipping a half table (Cyrillic a, Greek o) and calling it done. The confusables list is long and updated on revision. So the plan: fixture lands first, table gets pulled from source with the revision pinned, and the scanner reports that revision in its output so results stay reproducible. Thanks for closing the gap I only half saw.
Worth pinning the Unicode Character Database (UCD) version too, alongside the confusables.txt revision. NFKC normalization itself isn't stable across UCD versions -- new codepoints, changed decompositions -- so the same input string can normalize differently depending on which UCD your normalization library shipped with. If the scanner reports the confusables.txt revision but not the UCD version behind its NFKC step, two runs on different library versions can silently disagree on whether a given string trips the heuristic, even reading the same confusables table. Pin both, report both in the output.
Good catch, and it's the same reproducibility problem one layer down. I was treating confusables.txt as the only moving part, but if NFKC runs on whatever Unicode data ships with the Python build, two machines can disagree without either one being wrong. Python exposes that as unicodedata.unidata_version, so reporting it next to the confusables revision should be cheap. "Pin both, report both" goes into the fixture 3 plan as you wrote it. Do you know whether confusables.txt revisions are tied to a specific UCD version, or can the two drift apart?
They're published together in each Unicode release — confusables.txt is part of UTS #39, versioned alongside the UCD for that release — but that doesn't stop them drifting in practice, because they usually don't come from the same place at runtime. Python's unicodedata module bundles whatever UCD version shipped with that interpreter build, but if your fixture pulls confusables.txt fresh from unicode.org (or vendors it separately), you can end up pairing, say, UCD 15.0's unidata_version with a confusables.txt from 15.1. Same failure mode as the original bug: two 'authoritative' sources that happen to usually agree, until a release cycle where they don't. If you're pinning both, worth pinning them as a pair from the same Unicode release rather than independently — pin-drift between correctly-pinned-but-mismatched versions is the same silent divergence as not pinning at all.
@reidmarlow Sharpest framing of the miss I have read, and re-checking the AST after your comment added a nuance: Python folds adjacent string literals into a single Constant at parse time, so a pure AST visitor would already see the joined text. My scanner missed it because the rule scans raw source lines, not the AST: a phrase split across lines never appears whole in any single line, and no folding happens at the line-scan layer. Your lexer-level token join and my pre-join fix converge on the same rule: join before match.
The case where your literal_eval pass earns its keep is explicit concatenation, a BinOp like "foo" + "bar", where there is no adjacency for the parser to fold. That one goes on the fixture list next to the homoglyph case. Two new blind spots found in one comment thread, this is exactly why I ship the misses.
The concatenated string split is brutal for regex scanners. As soon as code authors or LLMs format long prompts as adjacent string literals across multiple lines, standard line-based AST visitors treat them as separate constant nodes unless you explicitly run a constant-folding pass.
AST visitors that inspect ast.Constant directly without constant folding miss this every time. A quick pass with ast.literal_eval on joined string expressions or tracking adjacent string tokens at the lexer level makes that fixture trivial to catch before touching full semantic analysis.
That matches the fix direction I wrote down, pre-joining adjacent string literals before matching, but you've given it a more concrete home. Working at the token level appeals to me because it skips a full AST pass and keeps the original line numbers for the finding. Would you fold only implicit concatenation, or also explicit + between two literals?
The homoglyph case is worth adding as a third obfuscation fixture, since it defeats both of your current normalization ideas differently than zero-width chars do. Swap a Latin letter in the trigger phrase for a visually identical Cyrillic or Greek one (a→а, o→ο) and the literal pattern misses for the same reason as the zero-width case, but stripping zero-width/bidi characters won't fix it — the character is present and rendered, just not the codepoint the regex expects. The general fix is NFKC-normalizing plus a confusables table (Unicode's own confusables.txt) applied to a canonicalized copy before matching — same "flag when canonical trips but original doesn't" pattern you're already using for the zero-width case, just a different normalization step. Worth adding now while you're already building the obfuscation fixture set.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.