A page of ours returns 200. The HTML validates. Every test is green. The h1 reads exactly what we wrote. And for several weeks it was shown to about 1,580 people a month in search results and clicked once.
One click. At an average position of 8.7, where the usual click-through is somewhere between one and two per cent, so the expected number was twenty or thirty.
Nothing was broken in any sense a test or a log would recognise. What was broken was the <title> element, and it was broken in a way that nothing inside the application could see.
One variable, two outputs
The controller builds a single string and hands it to both places that need it: the page title and the Open Graph title. One assignment, two consumers, same value.
What came out the other end was not the same value twice.
h1 (68): TypeError: Failed to Fetch - What It Means and How to Find the Cause
og:title (68): TypeError: Failed to Fetch - What It Means and How to Find the Cause
<title> (58): TypeError: Failed to Fetch - What It Means and How to Find
A view helper fits the title element to a character budget, cutting on a word boundary. That is a reasonable thing for a site to do and it was added deliberately. The point is not that the truncation is wrong. The point is that nothing at the call site says it happens, the other two consumers of the same string are untouched, and the result is a sentence that stops before its own subject.
That is the page with the 1,580 impressions. The half it loses, "the Cause", is the half somebody searching for the meaning of an error is actually looking for. Four of our highest-traffic pages were shipping a title that ended mid-phrase like this; another stops at "and the Keyboard Key", dropping the thing the article is about.
Why no test caught it
Because every test asserted on the value we set, not the bytes that left.
The model holds the full string. The controller holds the full string. The template is handed the full string. You can write a perfectly good test at any of those layers, watch it pass, and learn nothing, because the transformation happens below the lowest layer anybody thought to assert on.
This is a general shape and it is worth naming: for any output whose consumer is outside your process, the thing to assert on is the rendered bytes. Search results, email clients, social cards, RSS, webhook payloads, PDF exports. Every one of those goes through a layer that can reformat, truncate, escape or re-encode what you set, and every one of those consumers is somewhere your test suite is not.
The test that would have caught it is boring:
test "the title element ships the whole headline" do
get article_path(@article)
title = Nokogiri::HTML(response.body).at_css("title").text
assert_equal @article.title, title
end
It asserts on response.body, which is the only artefact that is actually true. Everything above it is an intention.
The expensive property
What makes this class of fault costly is not severity. It is that it produces no event.
There is no exception, no 4xx, no stack trace, no log line, no alert. The page is correct by every check that runs inside the system that produced it. Monitoring is green because monitoring measures whether the page was served, and it was.
The only evidence exists outside your application, in a number that somebody has to go and look at, in a tool they open occasionally, weeks after the change that caused it. We found ours because a weekly job finally ran after being unavailable for nineteen days, and the number was too strange to ignore.
Until then, every signal we had said the page was fine. Every signal was telling the truth about the thing it measured.
The shape worth taking away
A failure that your system cannot observe is not a smaller problem than one that throws. It is a longer one, because the clock starts when somebody outside notices and decides to tell you, and most of them do not.
That is true of a truncated title, which at least leaves a number behind. It is far more true of the things that leave nothing: a customer whose page half-loaded and who closed the tab, a user who hit an error and shrugged, a visitor who saw a broken image and assumed the site was like that.
Those are the same category. The system behaves correctly from the inside, the only witness is outside it, and whether you ever find out is entirely up to somebody who owes you nothing.
Top comments (0)