A 404 is the easy half. Scoring a package that actually exists is where we got it wrong.

Someone left this on my last post:

I'd also treat the "404 means stop" rule as only the first gate. A package resolving successfully doesn't establish that it is trustworthy. For newly registered packages that closely match common model-generated names, provenance and publication history become useful signals too.

The deeper lesson is that agents need pre-action verification, not just error recovery. Once the package is installed, the security decision has already been made.

Both correct, and the first one sent me to look at our own data. We publish a list of package names that LLMs invent. 1,885 of the 1,911 entries simply do not exist — those are easy. 26 of them answer 200.

I checked those 26 by hand. Almost all of them were wrong.

The one that stung

name        : opentelemetry-configuration
author_email: OpenTelemetry Authors <cncf-opentelemetry-contributors@lists.cncf.io>
repository  : github.com/open-telemetry/opentelemetry-python
summary     : OpenTelemetry Python Declarative Configuration (experimental)

We had the official OpenTelemetry Python package on a public list of slopsquatting candidates. Not because anything about it is suspicious — because it was 41 days old when we looked, and our rule said 14–60 days is medium.

The same rule caught cdx-rs, which is a cd replacement for the terminal:

crate  : cdx-rs
desc   : A fast and interactive cd command alternative for directory navigation
repo   : github.com/Cinnamon-jp/cdx
created: 2026-08-30

A model suggested cdx-rs as a Rust crate for parsing CycloneDX SBOMs. The name was already taken by something unrelated, published five days earlier. We scored it high.

Age was a proxy for a question I wasn't asking

The reasoning behind an age window is sound: an attacker who watches what models hallucinate registers the name afterwards, so a fresh registration under a hallucinated name is suspicious.

Read that sentence again and the actual signal is right there. Afterwards. Not "recently" — after the hallucination.

I had been measuring the wrong interval. created_at against today tells you how new a package is, which is a fact about the calendar. created_at against the first time any model produced that name tells you whether the registration could have been a response to it, which is the thing I actually wanted to know.

And we already stored it. Every finding carries hallucinated_by, with a first_seen per model, because the dataset is meant to be evidence rather than a count:

{
  "model": "gemini-2.5-flash",
  "task": "Rust crate for parsing CycloneDX SBOM",
  "first_seen": "2026-09-04T..."
}

cdx-rs was published 2026-08-30. First sighting 2026-09-04. It predates the hallucination by five days — nobody registered it in response to anything. The new rule is four lines:

if first_hallucinated and info.created_at < first_hallucinated:
    return "low", "registered before the name was ever suggested"
age_days = (now - info.created_at).days
if age_days < 14:
    return "high", "registered after the name was suggested"
if age_days < 60:
    return "medium", "young"
return "low", "established"

Both false positives drop to low and come off the list. A name registered after the first sighting keeps its score, and now the status string says why, which it never did before.

This is not airtight. Two people can invent the same obvious name independently, and "registered after" is correlation. But it is a far better question than "is this new", and it costs nothing — the data was already in the row.

npm's 200 means more than one thing

Three entries were labelled registered but never published: native-websocket, node-ics, sql-injection-detector. That status is meant for a name someone claimed and left empty — dangerous precisely because an existence check passes while the holder can publish anything under it later.

Except:

curl -s https://registry.npmjs.org/native-websocket | jq '.time'

{
  "created": "2020-11-29T...",
  "modified": "2020-11-29T...",
  "unpublished": { ... }
}

These were published and then taken down — in 2016, 2020 and 2022. npm leaves a tombstone: 200, no versions, and time.unpublished set. My check was versions == 0, which cannot tell an empty squat from a grave.

It matters because the two have different mechanics. An empty registered name is held by someone who can publish to it today. A tombstone is a name that was real — which is plausibly why a model learned it in the first place — and whether anyone can claim it depends on the registry's unpublish policy, not on the holder's intent. They are medium now, with a status that says what they are.

If you are doing this yourself: don't read versions alone. time.unpublished is the field.

A score is a statement about a moment

The last problem had no clever cause. opentelemetry-configuration was 41 days old when we scored it. It is 77 days old now. Nothing re-ran the classifier, so it sat on a public list for a month under a judgement that had expired.

Every finding that resolves is now reclassified on each cleanup run, and anything that drops to low is withdrawn — kept at its URL with a notice, because other people link to these IDs, but out of the list and out of the API.

The finding here is boring and general: if your scoring function reads a clock, something has to re-run it. A pipeline that only ever classifies on insert will accumulate verdicts that were true once.

What I still don't have

The comment's other word was provenance, and that is the honest gap. We score the name and the registry's timeline. We do not check who published it or from where.

Both big registries now expose this:

• npm publishes provenance attestations — a signed statement that a package was built by a specific GitHub Actions workflow from a specific repository.
• PyPI supports PEP 740 attestations on the same idea.

"Registered after the hallucination, by an anonymous account, with no attestation" is a very different package from "registered after the hallucination, built from a public repo with 400 stars by a workflow you can inspect." Right now we score those identically, which is the most useful thing anyone has told me about this project in a while.

That is the next signal. Until it lands, the list says what it can support and not more.

The data is here — CSV and JSON, daily, CC BY 4.0. The gate is one call and needs no key:

curl -X POST -H "Content-Type: application/json" \
  -d '{"packages": ["pkg:npm/fast-jsonwebtoken"]}' \
  https://vdb.ai.kr/v1/ai/check-packages

Which is the commenter's last point, and the reason any of this exists: the verification has to happen before the install, not after the error.