The last step of my release pipeline installs the package I just published, on Linux, Windows and macOS, and confirms it reports the version that was released. It is the only thing in the whole chain that tests the artifact a stranger would actually download, rather than the source it was built from. Last night it reported success on all three operating systems against a version the registry was not serving.
Here is the step, as it had been since the first release:
v="${TAG#v}"
for i in 1 2 3 4 5 6; do
out=$(npx --yes "$PKG@$v" --version 2>&1) && break
echo "attempt $i: $out"; sleep 20
done
echo "$out"
case "$out" in *"$v"*) echo "smoke ok: $PKG@$v" ;; *) echo "expected version $v in the output"; exit 1 ;; esac
Read the last line on its own. It passes when the released version appears anywhere in the output. Now read what npm prints when the version is not there:
npm error notarget No matching version found for ai-slop-linter@0.1.5.
The failure message contains the version. So when the registry was still processing the publish and all six attempts failed, $out held that error, the pattern matched it, and the job printed smoke ok and went green. Three operating systems, three green checks, nothing installed on any of them.
Nothing was broken underneath. The package had published correctly and npm was simply slow to serve it, which it warns you about in its own output: Your package is being processed and may take a few minutes to become available. So the check did not hide a bad release. What it revealed is worse in a quiet way. It could not have caught one. The success condition was contained in the failure output, which means that for as long as that pipeline had existed, its final verification had been decorative.
I found it because I was watching the registry by hand while waiting for a version number to change, and the number would not change while the workflow insisted it had. Without that accident it would still be there.
Five more of the same shape, in one night
That was not the only one. It was the third of six found in about four hours, across the same set of repositories, and I want to list them because the pattern only becomes obvious as a set.
A version checker that knew four phrasings. I keep a script that compares every version number written in my documents against what npm and PyPI actually serve, because all five of them were wrong at once, on a page that said version numbers were deliberately kept off it because they would rot. The script knew four ways a version can be written in a sentence. A table cell writes a fifth: the package name sits in the first column and the version three columns away. So npm 0.1.1 in a document that goes to an assessor matched nothing, and the script printed ok, every claim in 6 surfaces matches.
When I reintroduced the two wrong numbers to see what the old version would say, it reported neither of them, and instead demanded a correction to the one number in the whole pack that was right: a deliberately frozen snapshot dated 5 September, which it wanted raised to today's release. That is not a hypothesis about how the snapshot had been corrupted. It was the corruption happening again on demand, and it explained a line I had been puzzled by, a pairing of an old npm version with a current PyPI one that had existed on no day in either registry's history.
A checker reading the right file at the wrong moment. Another script guards the claims on a public page about my published packages. It read the conformance table from the default branch. Every other claim on that page is about the version people install. So the moment a pull request merged, it called a correct sentence stale. It now reads the table at the tag of the released version.
A guard present on two claims and missing on the third. The same file already had two match-nothing guards, added after that exact failure had cost something twice. The third claim, the one comparing npm against PyPI, had none. Reword the sentence and the comparison silently stops running, with the file reported clean. Knowing the lesson is not the same as having applied it to every claim in the file.
Five tests that failed for the wrong reason. Publishing three articles meant adding one row to a data file. Five tests broke, none of which was testing publication: the homepage's first link, a feed's build date, a pagination assertion, a static route list, and a fixture guard whose message read "twelve items (eleven essays plus one external piece)". Every one of them had a title, a URL, a date or a count typed into it. The repair that suggests itself is to retype the literals, which restores the green and the defect together, and buys the identical morning on the next publish.
And two numbers I wrote from memory. Writing the record of all of this, I described a cache lag as lasting "about twenty minutes". I had not measured it. Neither had the reviewer who had written "hours" for the same interval. Both of us invented a figure inside a document about figures that arrive by inference and get recorded as measurements. The defensible version, from timestamps: the three articles published at 21:25:31Z, 21:26:28Z and 21:27:15Z, and the index was serving all seven by 21:37:47Z. How long it had been stale before that is not something either of us recorded, and the record now says so instead of picking a number.
The question I had never asked of a check
Six things, and they look unrelated. A shell pattern, a regular expression, a git ref, a missing guard, five hard-coded literals, two remembered intervals. They are one defect.
Every one of them reported agreement for a reason that had nothing to do with what it was checking. The version checker found nothing because it did not know the sentence shape, and silence rendered as success. The conformance checker read a real file at a moment when that file was not the subject. The smoke test read a real string in output that said the opposite. None of them failed, and none of them could have, for the specific thing they existed to catch.
The question I had been asking when I wrote each of these was "does this find the problem". I had tested each one against a broken input at the moment I wrote it, in the sense that I had convinced myself it would fire. The question I had never asked is the one that matters:
What does this do when it finds nothing, and is that distinguishable from finding everything in order?
A check whose answer to that is "they look the same" is not a check. It is a comment that runs.
The cheap test follows directly. Before trusting a clean result, run the thing against a deliberately broken input and confirm it fails, and that it fails for the right reason. It takes a minute. I did it for each repair last night: reintroducing the two wrong versions so the old checker could be caught calling them fine, and running the new smoke condition against the real npm failure text. For the shell one the whole fix is to stop testing a substring and start tracking two facts separately:
installed=""
for i in 1 2 3 4 5 6; do
if out=$(npx --yes "$PKG@$v" --version 2>&1); then installed=yes; break; fi
echo "attempt $i did not install:"; echo "$out"; sleep 20
done
[ -n "$installed" ] || { echo "the registry never served $PKG@$v"; exit 1; }
printed=$(printf '%s' "$out" | tr -d '\r' | tail -n 1 | tr -d '[:space:]')
[ "$printed" = "$v" ] || { echo "installed, but --version printed '$printed'"; exit 1; }
Did it install, and did the thing it printed equal the thing we released. Two questions with two answers, neither of which the failure message can satisfy by accident.
The direction that looks like diligence
The five broken tests belong to the same family, seen from the other side, and they are the more dangerous half.
A test pinned to a literal does not pass for the wrong reason. It fails for the wrong reason. Five red tests after adding one row of data feel like a suite doing its job, and that feeling is what makes the trap work: the fix that suggests itself is to update the literals, which is fast, restores the green, and leaves the defect exactly where it was. A silent green at least feels like something you ought to verify. A loud red feels like something you ought to fix, and fixing it is not the same as understanding it.
Those five now derive from the same function the pages derive from, and assert properties rather than snapshots: that the first link on the homepage is the newest item, that the last page is whatever page the count implies rather than page two. The person who rewrote them also checked both against deliberately broken sources, a reversed slice and a clock-derived build date, before trusting the green. That is the rule applied to the repair rather than only to the thing repaired, which is the step everyone skips, me included until that night.
Why writing it down worked when re-reading did not
The part I keep coming back to is not any of the six. It is how the last two were caught.
The work was reviewed across two sessions, with review passing between them; the repositories disclose that the work is AI-assisted, and this is the shape of that assistance rather than the subject of this piece. Three of the six were found by one reviewer looking at the other's work, which is unremarkable. The last two were each of us catching our own wrong number, which sounds like carefulness and is not.
Neither of us caught our own figure by re-reading it. I caught mine because a question forced me to state it precisely enough to be compared against a timestamp. The other was caught because a correction in the identical shape had just been written down next to it. Neither correction started with suspicion. Both started with a claim that had been recorded somewhere specific enough that comparing it to reality was a thirty-second operation rather than a project.
That is a property of the record, not of the people keeping it, which is the only reason it is worth writing down. Carefulness does not survive a long night. A claim written so it can be checked will sit there being checkable whether or not anyone is currently suspicious, and it will still be checkable next week when everyone has forgotten which parts were measured and which were remembered.
All six were claims that had been written somewhere and never re-run. A table cell, a conformance table, a shell condition, an unguarded pattern, five test literals, and two intervals. Not one of them was found by noticing that something felt wrong. Every one was found because somebody eventually compared the written thing against the thing it described.
The smoke test is the one I would keep if I could only keep one, because it is the purest form: a check whose success condition was contained in its own failure message, sitting at the end of a release pipeline, printing smoke ok on three operating systems for as long as the pipeline had existed. If that can happen in a repository where every number in the documentation is compared against a registry by a script on a schedule, it can happen anywhere.
I write about the engineering decisions behind a proactive assistant and five small open-source packages at efe-genc-portfolio.vercel.app, where this piece is the canonical copy. If you have one of these six in your own pipeline, or you think the rule is wrong, I would rather read the disagreement than the agreement.
Top comments (18)
I recognise the two numbers written from memory most. I keep one facts file where every figure carries the date it was last true, and a number does not go into anything I publish unless it is in that file. Anything retyped from memory counts as unverified.
The table cell with the version three columns away is a good example of the other problem. A checker written against sentences reports clean on every format it never learned to read.
What stuck for me was making the snapshot's age its own failure condition. If the source has been written since the snapshot was taken, the run stops and says unknown rather than comparing anything.
A checker that can only report match or mismatch has no way to tell you it was reading yesterday.
That is a better rule than the one I shipped, and I have implemented it.
Mine bounded the snapshot's age at six hours. Your version bounds the thing the hours were standing in for. Six is arbitrary: a five-hour-old snapshot taken after the last change passes, a seven-hour-old one taken after nothing passes nothing. The hours were a proxy for "has anything moved since", which is uncomfortable to notice in a piece whose whole subject is proxies standing in for conditions.
I could only take half of it, and the half matters. Most of what my snapshot holds is remote: npm, PyPI, the GitHub API. I cannot tell whether those moved without collecting again, so the hour bound stays for them, now labelled in the code as the proxy it is. But one source is local and comparable directly, so a commit in the repository newer than the snapshot is now its own failure condition:
Exit 2, same as staleness, so the commit hook refreshes instead of blocking.
Tested in isolation rather than assumed, because the age bound would otherwise mask it: with the hour bound lifted, a snapshot older than the last commit exits 2 with that message, and a snapshot newer than the last commit exits 0 even at 30.9 hours old. The control is what makes the first result mean anything.
"No way to tell you it was reading yesterday" is the sentence. A checker with two outcomes cannot report the third thing that is true surprisingly often, which is that it does not know.
Good call labelling the hour bound as a proxy in the code. Whoever touches it next will know what it is standing in for.
For the remote side, npm and the GitHub API both return an ETag, so a conditional request can often tell you whether anything moved without collecting it all again.
Thanks, that could make the remote check much cheaper. I’d keep the cached response and its ETag together, then use If-None-Match on the same endpoint to revalidate it.
I’d also keep “collected at” separate from “last validated”: a 304 can confirm that particular response is unchanged, but it does not mean the local tests have run again or the whole evidence snapshot is fresh. Checking support on the exact npm and GitHub endpoints the collector uses seems like the right next step.
On the GitHub side that also saves the rate limit. An authenticated request that comes back 304 doesn't count against it, so revalidating on every run costs next to nothing.
You are right, and I measured it on the endpoints the collector actually calls rather than on the services in general. GitHub:
x-ratelimit-usedwas 13, the conditional request came back 304 reporting 13, and the next unconditional call reported 14. One of the two was charged.npm turned out to be the awkward one, and not in the direction I expected. The endpoint I use,
/<name>/latest, returns no ETag at all: onlyLast-Modifiedandmax-age=300. The full package document does return one. So the small response is the one with no validator, and the large one is the one I could skip. The choice is between always fetching something small and usually skipping something big, which is a question about how often the answer changes rather than about bytes.If-Modified-Sinceis available on/latest, so that is the cheaper thing to try first. PyPI returns an ETag.It also settles the field question for me. A 304 says one response is unchanged; it does not say the snapshot was collected again, so "last validated" has to be its own field rather than moving "collected at" forward.
Shipped, and the measurement is better than I expected. Sixteen of the collector's seventeen remote reads now revalidate: ten GitHub reads and PyPI with
If-None-Match, five npm reads withIf-Modified-Since. A cold run movesx-ratelimit-usedby ten and a warm run by zero, measured from the response headers rather than from the docs.Three things I would not have guessed. npm's
/latestcarries no ETag and does carryLast-Modified, while the full package document carries a strong one, so the cheap endpoint is the one with the weaker validator.gh api -iexits non-zero on a 304, which means the exit status cannot tell a revalidation from a failure and the status line has to be read from stdout with stderr kept apart. And the outbound search sendsCache-Control: no-cachewith no validator at all, so it is collected in full every run and the snapshot records that instead of implying otherwise.Two fields, not one:
collectedAtfor when it was gathered andrevalidatedAtfor when every source last confirmed. A source that errored is unknown rather than unchanged. Five tests hold the rules, including a 304 arriving with nothing cached to validate.Keeping an errored source as unknown instead of unchanged is the detail I would have missed. A failed revalidation that reads as nothing moved brings the stale snapshot straight back, and nobody notices because the run still looks green.
The npm one is a good trap as well, the cheap endpoint carrying the weaker validator.
The 2>&1 in that npx line is doing quiet damage too: it merges npm's error output into the same variable the case statement reads, so the failure text is handed straight to the matcher that decides success. Without it, $out would have been empty on a miss and the pattern could not have matched. The general shape I'd take from this is that a check whose success string can appear in its failure output isn't strict, it's unfalsifiable. Assert on the exit status first, then on something only a real install can produce.
Thanks for running it rather than nodding - 346 bytes and ETARGET quoting the version straight back is a better demonstration than the argument was. One thing your loop already has that the verdict throws away: out=$(...) && break consults the exit status, and then the case statement decides on text alone. Capturing stderr separately and gating on the status first means npm's wording can change next release without your check changing its mind.
You were right, and the stub showed how far it goes. The install half already gates on the exit status, so that part holds. The comparison that runs after it reads the same variable, and
2>&1had put stderr inside it.Measured on the block itself rather than on a rewrite of it. I extracted the
run:body fromrelease.ymland ran it against a fake npx, four scenarios:printed=npmnoticeand failed a release that had installed and printed 0.4.2 correctlySo what it was hiding points at a false red, which is the safer of the two directions, although a red that nobody believes is how the original bug survived six retries. stderr now goes to a temp file and is printed on every failing path, so the log keeps everything it used to show.
Shipped in the four repositories that share this workflow: workproof#38, ai-slop-linter#36, proactive-gate#47, surviving-lines#21. Your handle is in the commit and in the comment above the loop.
The update-notifier case is the one I would have missed, and it is the worst-shaped failure in that set: the notifier only writes on its own check interval, so the red is intermittent and a rerun clears it. That is exactly the shape that teaches a team to press retry instead of reading the log, which is how the original bug survived six of them. Extracting the run: body and driving it against a fake npx is the part I would steal - testing a rewrite instead of the block is how you end up verifying a workflow that is not the one running. Does the temp file get printed on the success path too, or only on the failing ones?
'Its final verification had been decorative' is the sentence that stings — a smoke test that can only reproduce what the build already believes is a mirror, not a check. The registry being the one source a stranger actually hits is exactly why it deserves its own probe, not the pipeline's opinion of itself.
The five tests that broke on one data row is the detail I'd underline: when a homepage link, a feed date and a fixture guard all hardcode the same fact, green means 'nothing changed', not 'everything is right'. The repair that retypes literals restores the green and the defect together — I've bought that identical morning more than once. What finally held for me was deriving assertions from the data file at test time, so adding a row can only break a test that genuinely reads it.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.