My filter matched 0 of 1545 rows in production, and the test fixture was the reason nobody noticed

A feature was shipping nothing. Not erroring, not slow — the list it was supposed to populate was empty, for everyone.

The cause was a filter matching on a string prefix:

def is_actionable(rec: str) -> bool:
    return rec.startswith("FIX:")

In production, 0 of 1545 rows started with FIX:. The prefix had been dropped upstream months earlier, in a change that had nothing to do with this code and looked entirely safe.

The test that passed

def test_is_actionable():
    assert is_actionable("FIX: add a canonical tag")
    assert not is_actionable("INFO: nothing to do here")

That test is green, and will stay green forever, and tells you nothing.

Because I wrote the fixture by reading the filter. The string "FIX: add a canonical tag" exists in my repository for exactly one reason: the function I was testing looks for FIX:. It is not a sample of anything. It's the function's own precondition, restated as data and then asserted on.

So the test verifies that startswith works. Python already had that covered.

The shape of the bug

The filter has two halves and only one of them is in the code:

1. Given a string with the prefix, do the right thing. In the function. Tested. Correct.
2. Production strings have the prefix. Nowhere. Untested. False.

Half two is an assumption about a different system — whatever produces those records. When that system changed, nothing in my repository was watching, because nothing in my repository contained a real example of its output.

A fixture written from the code under test can only ever check half one. That's the whole failure mode, and it's invisible from inside the test file, which reads as perfectly reasonable.

What I added

Not more unit tests. A test on the population.

def test_filter_matches_some_real_records(db):
    total = db.count("records")
    assert total > 0, "no records to check against"
    matched = db.count("records", where=is_actionable)
    assert matched > 0, (
        f"is_actionable() matched 0 of {total} real records — "
        f"the shape it expects no longer exists upstream"
    )

Three things about this that took me a couple of tries.

assert total > 0 first. Without it, an empty table makes the second assertion vacuously... well, it makes it fail, which sounds fine, but the message is then misleading — it says the filter is wrong when the fixture is empty. Guard the denominator and the failure tells you which of the two things broke.

matched > 0, not a percentage. I wanted matched / total > 0.1 and I was wrong to. The ratio is a business fact that legitimately drifts, and a threshold on it produces a test that fails on a good day. Zero is the only number that unambiguously means the contract is gone.

It has to run against real shapes. A sample of production data, an anonymised snapshot, a staging DB, or the smallest thing that works: a handful of real strings copied out of production into a fixture file, with a comment saying where they came from and when. Not strings you wrote — strings the upstream system wrote.

The rule I took out of it

A fixture derived from the code under test can prove the code is self-consistent. It cannot prove the code is right about the world.

Every filter, parser, regex, or mapping that consumes data from somewhere else has this second half, and the second half is where the outages live — because it breaks without anyone touching your file.

The cheap version, if you add nothing else: one assertion, anywhere in your suite, that the filter matches at least one real record. It catches the entire class, and unlike a unit test it fails when the world changes rather than when you change.

How I'd find the next one

Grep your codebase for string-shape assumptions about foreign data — startswith, endswith, a hardcoded key, a regex with a literal prefix, an enum comparison against a string from an API:

grep -rnE '\.startswith\(|\.endswith\(|re\.(match|search)\(r?"[^"]*\^' --include='*.py' .

For each hit, ask the question that isn't in the test file: what proves that shape still arrives? If the answer is a fixture you wrote, you have this bug and it's dormant, not absent.

Mine was dormant for months. The tests were green the whole time, and green tests are how it stayed dormant.