I published a survey of 4,749 public MCP server schemas a few weeks ago. The headline was that 17.7% of tools carry a description containing no word that distinguishes them from a sibling tool on the same server.
Then people who actually run MCP servers showed up in the comments, and over about ten days they took the measurement apart.
One found a bug. Several found a design flaw. Two of them found opposite design flaws that cancel each other out, which was the most useful thing that happened. And one found a ceiling the whole approach can't get past.
Here's all of it, because the corrections are worth more than the original number.
The bug: no stemming
The metric counts content words in a tool's description that appear in no other tool's description on the same server. It does exact string matching.
Someone pointed at two descriptions from his own server:
analyze_job_description "Extract what a job posting actually screens on."
optimize_resume "Rewrite a resume so it passes ATS screening."
screens and screening. Same word, two forms, and my counter treats them as unrelated — so these two tools score as distinctive on the one word that actually links them.
That's not a nuance. It's wrong, and it's wrong across all 82,549 tools. Stemming before counting is a small change with an unknown effect on the headline figure, which I'll report when I've re-run it.
The design flaw: lexical distinctness isn't ambiguity
The same author gave me his full ten-tool list and said three of them were confusable in practice. So I ran the metric on it.
50.0% score_resume "Score a resume for ATS compatibility."
60.0% analyze_job_description "Extract what a job posting actually screens on."
60.0% optimize_resume "Rewrite a resume so it passes ATS screening."
75.0% search_jobs "Return the user's job matches."
100.0% update_job_preferences "Set the roles and locations it hunts for."
His ambiguous cluster — score, analyze, optimize — comes out at 50–60%. My corpus calls that healthy.
My first fix was to compute distinctiveness on verbs alone, on the theory that verbs carry the action and nouns are shared boilerplate.
That fails harder. His ten verbs are score, extract, rewrite, produce, write, translate, return, set and build. All distinct. Verb-only distinctiveness scores every tool at 100%, including all three ambiguous ones.
The problem isn't granularity. Lexical distinctness and semantic distinctness are different properties, and no word-counting metric at any resolution closes the gap.
What does show the cluster is the shared input nouns — resume in four descriptions, job in three, ats in two. Those three tools are ambiguous because they act on the same object, not because they're worded alike.
His rule, better than my number: if two tools can ever be correct for the same sentence, you don't have two tools. You have one tool with a parameter.
So the fix isn't a better distinctive share. It's a second signal, measuring something else entirely. Distinctive share asks whether two descriptions look alike. Input overlap asks whether two tools could both be right.
The correction to the correction: overlap alone flags good design
Someone running a six-tool ticket server pushed back, with a pair he'd deliberately kept:
Batch retrieval and an analysis bundle. Identical input shape. Different intended output — flat details for more tickets, versus bounded evidence with optional comments and source anchors.
Input overlap would flag that pair. He'd be right to dismiss it, because the boundary is stated: a model reading those two descriptions has something to discriminate on even though the inputs are the same.
So overlap needs a partner. Overlapping inputs with distinguishable outputs is fine design. Overlapping inputs with indistinguishable outputs is the failure. Flagging the first is exactly the false positive that gets a signal switched off — and on a large server, a 10% false positive rate buries everything.
His test is better than any metric I can compute: can a model infer the intended output and boundary from the name, description and parameter contract alone, without knowing the repo or the author's intent?
The two corrections that fight each other
This is the pair I'd have got wrong silently.
One reader pointed out that the metric penalises focused servers. A memory server says "memory" in every description by design. A ticket server says "ticket". Those words count as shared, but they aren't a collision — they're the name of the server, repeated. Drop words appearing in more than 80% of a server's tools before counting, and small focused servers score higher.
Correct, and I was about to implement it globally.
Then the ten-tool author showed why that breaks the other signal. On a focused server, the collision is the domain noun. Strip resume from his descriptions and you delete the exact words that make score, optimize and analyze confusable. I'd have shipped a sharper distinctive-share number that quietly destroyed input overlap.
The resolution is two readings with different preprocessing, not one corrected score:
- Domain-stripped distinctiveness — do these descriptions look alike?
- Domain-intact input overlap — could two tools be right for the same sentence?
And the gap between them is the finding. A server that scores high on the first and badly on the second is focused, acts on one object throughout, and never says what its tools return. That's a common shape and it deserves its own line rather than being averaged away.
A third reader then improved the threshold out of existence. Frequency isn't a dial to tune — it's a distinction the same count already makes. A noun in nearly every tool names the server. A noun in a subset of tools names a cluster, and that subset is your candidate collision list.
Run it on the ten-tool server and the tightest group is ats, shared by score and optimize and nothing else. Which is the pair he flagged first. Same counter, no threshold, no tuning.
The ceiling
The same reader then found the thing none of this reaches.
analyze_job_description and score_resume share no word at all, stemmed or not. And they still collide, because "check my resume for this job" names both objects at once. The tools partition an intent that the sentence doesn't.
That collision exists between a request and a set of tools, not between two descriptions. Nothing computed on descriptions alone will ever see it.
That's a hard ceiling on static analysis, not a gap to close with a cleverer metric. The only artefact that catches it is a small labelled set of real requests — a handful of plausible sentences and which tool each should route to. Which is the one thing a corpus can't generate.
Two things I hadn't measured at all
Names. Two people independently said that when descriptions collide, models fall back to pattern-matching on tool names. One put it: "the model stops reading descriptions and pattern-matches on names, and then your error rate is really a naming problem."
That changes what a collision means. Gmail_DeleteDraftEmail and Gmail_SendDraftEmail have zero-distinctive descriptions — that pair was my headline example — but their names are clear. The names are carrying the load. A hypothetical pair with the same descriptions and names like delete_item and remove_item scores identically and is in far worse shape.
Annotations. MCP has readOnlyHint, destructiveHint, idempotentHint, openWorldHint. Two readers raised the same point: authors set them and write "Read-only" and "REPLACES" in the prose by hand, because the annotations are read by the client deciding whether to prompt for confirmation, while the description is read by the model deciding whether to call the thing. Two readers, two channels, and nothing requires them to agree.
Which makes a mechanical check available: descriptions mentioning constraints with no matching annotation set, and annotations set with nothing in the prose. The first means the client can't protect the user. The second means the model doesn't know a tool is destructive at the moment it's choosing.
What I'd take from this
Every correction came from someone who knew what the text was for.
I measured a property of the text and treated it as a proxy for ambiguity. Lexical distinctness looked like a proxy until someone showed me ten tools where it isn't. Populated descriptions looked like described parameters until someone pointed at one that says nothing. Descriptions looked like the whole interface until two people said names do the work when descriptions fail.
The original study was careful about one thing: it drew a hard line between what a schema shows and what models do with it, and refused to claim the second. That line held. What didn't hold was the assumption that measuring the text well is the same as measuring the thing that matters.
The published numbers stand as published — they measure what they say they measure, and they're all floors. The more useful metrics are the ones nobody had asked for yet.
Original study, data and analysis scripts: https://github.com/getmcpulse/mcp-schema-study
The schema checker runs these measurements on your own tools/list in the browser. The corrections above are being added to it.
I build MCPulse, an SDK that reports what models actually do with your tools under real traffic. If something here matches what you've seen on your own server, I'd like to hear it — every improvement above came from exactly that.
Top comments (8)
Publishing the corrections as their own piece is the right move — the stemming bug and the "lexical distinctness isn't ambiguity" flaw are both more interesting than the original 17.7%. The second one matches what I see in practice: the tools that actually confuse a model aren't the ones with overlapping words, they're the ones with overlapping decision boundaries.
score_resumevsanalyze_job_descriptionread as distinct lexically but a model routing a request has to guess which one "evaluate my resume" means, and description text doesn't disambiguate that. The thing that's helped us most isn't better descriptions — it's measuring the confusion empirically: give the model a set of realistic user intents and see which tool it actually picks, then look at the wrong picks. That surfaces the real ambiguous clusters regardless of wording. Your verbs-only experiment is a clever proxy but I suspect it'll still miss cases where two tools share a verb and a plausible object. Have you considered a behavioral metric — pick-rate divergence across paraphrased intents — as the ceiling the lexical approach can't reach?Pick-rate divergence across paraphrased intents is exactly the ceiling, and naming it as a metric rather than a test is the part I hadn't got to. I'd been thinking of behavioural testing as ground truth to validate the lexical signals against — a pass/fail per pair. Divergence across paraphrases is a continuous measure of the thing itself, which is better, because it distinguishes a pair that's genuinely 50/50 from one that's stable until you rephrase slightly.
That second case is the one I'd most want to catch. A pair where "evaluate my resume" reliably picks score and "check my resume" reliably picks analyze isn't ambiguous to the model, it's ambiguous to the user — and no amount of rewriting the descriptions fixes a distinction the requester didn't intend to make.
To be clear about the verbs-only experiment: it's dead, not a proxy I'm still running. I tried it on ten real tools and all ten verbs were lexically distinct, so it scored every tool at 100% including the three the author had flagged. Your prediction about shared verb plus plausible object would have been the next failure if the first one hadn't already killed it.
What I've replaced it with is input overlap — do these tools act on the same object — gated on whether their stated outputs differ. That survives the ten-tool case. It does not survive score_resume versus analyze_job_description, which share no word and collide anyway. So the lexical ceiling is real and I've stopped trying to engineer past it.
The open question for me is whether the static signals are worth keeping once a behavioural metric exists. My guess is yes, as a cheap prefilter — you can't paraphrase-test 191 tools pairwise, but you can rank pairs statically and spend the model budget on the top fifty. Which makes the useful comparison not "which approach is better" but how much of the behavioural result the static signals recover. That's the number I want and don't have.
The schema-measurement saga is a great example of something we keep hitting from the server side: agents will repeat whatever a tool returns with total confidence, so the honesty of the response shape matters more than the polish. We build a booking-verification MCP (bookingtruth.com) and ended up making "the API must be able to say UNKNOWN" the core design rule - verdicts are graded by evidence tier and nothing below supplier-authoritative evidence can be called CONFIRMED.
Genuine question since you clearly think about MCP response quality: would you trust an external verdict service for a claim your agent makes to a user, why or why not? We are collecting both answers from builders this week.
Yes, with one condition that I think is harder than the accuracy problem: the uncertainty has to survive the hop.
Your design rule is the right one. But UNKNOWN only means something if the consumer can't flatten it, and a model reads your response the same way it reads your description — as text, with no provenance and no type system. If CONFIRMED and UNKNOWN arrive as two strings in the same
verdictfield, they're symmetric to the thing reading them, and the model's summary of an UNKNOWN can easily come out as "I checked and it's fine."So what I'd want isn't a more accurate verdict, it's a response shape where the degraded case is structurally different rather than lexically different. Not the same field with a different value. A different shape the model has to handle — the evidence tier as a required sibling, or CONFIRMED and UNCONFIRMED as genuinely different objects.
That connects to something from my own data. Empty results are the failure nobody reports: the call succeeded, returned nothing usable, and the protocol calls it success. An UNKNOWN has the same structural signature. It's a successful response with no actionable content, and it will be treated like one unless the shape stops it.
The second half of trust is chain of custody, and it's the part I'd actually hesitate on. If my agent tells a user something is confirmed, I own that claim, not you. So I'd want the verdict to carry its evidence tier all the way to the surface rather than being collapsed at my end — which means my prompt and my UI have to preserve it too, and most don't.
One measurable proxy, if you want a number rather than opinions: whether models retry after an UNKNOWN. A model that gets UNKNOWN and immediately calls again with different arguments hasn't accepted the answer — it's treating your honest non-answer as a failed call and looking for a better one. That's visible from your side, and it'd tell you whether your UNKNOWN is landing as information or as an error.
This is the sharpest version of the critique we've heard, and it's fair: two strings in one field are symmetric to the reader. What exists today is half of your ask - the verdict object carries evidence_tier as a required sibling integer (never prose), an authoritative boolean, indicated_state for the "looks like X, not proven" case, and a proof_url to a citable packet. But you're right that a model can still read state:"unknown" and paraphrase it as "checked, it's fine" - nothing in the shape stops the summary layer from upgrading it.
The structural-difference idea is now on our design board. Options we're weighing: the proof packet exists only on supplier-authoritative verdicts, so an UNKNOWN structurally lacks the artifact and there's nothing to cite when it gets upgraded; and distinct required-field sets per state, so a flattened read fails validation downstream instead of passing silently.
Your retry-after-UNKNOWN proxy is the part we'll steal first. It's measurable from our side: same-agent repeat calls with mutated arguments after an UNKNOWN mean the non-answer landed as an error, not information. We'll start counting it - if UNKNOWN is being treated as a failed call, the shape changes, not the docs.
On chain of custody: agreed, and it's why the proof packet is a citable object rather than a paragraph - the consumer can surface it verbatim instead of summarizing. The remaining gap is the one you named: our packet can survive our API, but the consumer's prompt and UI still have to choose to carry it.
The proof packet existing only on authoritative verdicts is the stronger of your two options, and I think it's stronger than the reason you gave. Distinct required-field sets make a flattened read fail validation downstream — but that only helps consumers who validate, and a model paraphrasing your response into prose never touches a validator. The absent artifact works on the model directly: there is nothing to cite, so a summary claiming confirmation has to invent the citation rather than merely omit one, and models are noticeably more reluctant to do that.
That also makes the failure observable to you. If a consumer reports a confirmation against a verdict that shipped no packet, you can prove the upgrade happened. Validation failures happen silently in someone else's stack.
Worth doing both, since they catch different consumers. But if you only get one, the absent artifact is the one that works on the layer that actually does the flattening.
On the retry proxy, one refinement now I've thought about it more. Same arguments repeated isn't a retry — that's polling or pagination, and counting it as a retry grades every well-behaved repeat as a failure. It has to be different arguments within a short window. Mutated arguments after an UNKNOWN is the model rewording the question, which is exactly the "I don't accept that answer" signal you want.
The interesting secondary number is what it mutates. If it retries with a broader query, it's looking for any evidence. If it retries with a narrower one, it thinks it asked wrong. Those suggest different fixes — the first says your UNKNOWN reads as "not found", the second says it reads as "bad request".
And your last point is the honest limit for both of us. You can make the packet citable and I can measure whether the model accepted it, and neither of us can make the consumer's UI carry it to the user. That gap is where the claim actually gets made, and it's outside both our processes.
Both points taken, and the refinement is right. Retry counting from the start here: mutated arguments within a short window only - same-args repeats are polling, not rejection. The direction taxonomy is the part we hadn't seen: a broader rewrite means UNKNOWN landed as "not found", a narrower one means it landed as "bad request". Those point at different fixes, so we will classify the mutation, not just count it.
On the absent artifact: agreed it's the stronger option, and the observability argument is what settles it for us. A confirmation claimed against a packet-less verdict is a provable upgrade we can detect; a validation failure dies silently in someone else's stack. Current behavior ships a proof_url on every verdict, UNKNOWN included, so adopting this means cutting the packet from non-authoritative verdicts in the next rev. That's now the leading option on the design board, with per-state required fields as a second layer for consumers who do validate.
And yes - the UI gap is the honest limit for both of us. We can make the packet citable and measure whether it was accepted; we can't make anyone's surface carry it. That gap deserves a post of its own.
Both shipped today. CONFIRMED verdicts carry the proof packet; non-authoritative verdicts ship no packet at all - a consumer claiming confirmation against an UNKNOWN now has to invent the citation rather than omit one. The retry counter is live too: mutated-argument retries within a 10-minute window after UNKNOWN, same-arguments excluded as polling, direction bucketed broader/narrower by whether the rewrite widens the evidence set. First internal numbers once there is enough traffic to read. The interesting open question is which misreading dominates - "not found" or "bad request".