My Scheduled Agent Ran 40 Times and Did Nothing — Here's the Assertion That Fixed It

I found this one by accident, which is the only way you find it.

Every night at 02:30, a job pulls new filings from an API, normalizes them, and drops them in a queue that feeds my morning briefing. It had been running for about six weeks. Every run: exit code 0. No errors in the log. The scheduler dashboard showed forty consecutive green runs. If you'd asked me, I'd have said that job was my most reliable piece of automation.

Then I noticed my briefing had gotten thin. Not empty — thin. Some days two items, some days none, and I'd been assuming "slow news." I finally got suspicious, went to the log, and realized I had absolutely nothing to be suspicious with: the log said fetched items, wrote to queue on every run, and that's it. No counts. No IDs. Just a sentence I'd written myself, once, and never verified.

So I added one line — a count — and reran it manually.

Zero. Every night. Forty times.

The HTTP call returned 200 with an empty array, because a parameter name in the query had been renamed on their side (since → from_date) and the API's response to an unknown parameter was to cheerfully ignore it. My for loop iterated over an empty list and completed successfully, which is exactly what a for loop over an empty list does. Nothing raised. Nothing retried. Nothing was logged as a problem, because as far as every layer of my stack was concerned, nothing was a problem.

That's the failure mode I want to write about, because I'd covered the other ones: I have a circuit breaker for hard failures, a health monitor for agents that are dying, and dedup for jobs that fire twice. All three of those watch for something happening. None of them watch for something not happening. And "not happening" doesn't throw.

Errors page you. Silence doesn't.

The reason this took six weeks is structural, not lazy. An exception has a moment: it fires, a handler catches it, an alert goes out. A silent no-op has no moment. It has a shape over time, and only if you're graphing the right thing.

My monitoring was graphing the wrong thing. I was tracking run status (success/failed) and run duration. Both were perfect. What I wasn't tracking was the only number that mattered: effects produced. Runs completed. Effects were zero. I had a dashboard that was green because it was measuring the system's opinion of itself.

The fix isn't more logging. It's a different question: not "did it run?" but "did it change anything?"

Rule one: assert the effect, not the attempt

Every job I write now has to declare what a successful run looks like in terms of side effects — rows written, messages sent, files created, IDs emitted. Then the job asserts it before it can report success.

def assert_effect(name, produced, expected=">=1"):
    if expected == ">=1" and produced < 1:
        raise JobProducedNothingError(
            f"{name}: ran fine, produced {produced} effects"
        )
    log.info("%s: effect ok (%s)", name, produced)

The point isn't the helper. The point is that a job can now fail for the crime of doing nothing. Before, "did nothing" and "did everything" were the same outcome — success — which means my success signal was measuring the wrong universe.

Two things I had to get right, and got wrong first:

An assertion that can't fail isn't an assertion. My first pass was assert len(rows) >= 0. That's a tautology wearing a safety vest. It passed on the empty list and I felt protected for another two days. The assertion has to be able to actually fail on the thing you're worried about — which sounds obvious until you write one at 1 AM.

Assert on the effect, not the attempt. resp.status_code == 200 is an attempt check. len(written_ids) > 0 is an effect check. The whole incident was a case where every attempt check passed and the effect was zero. If your assertion can be satisfied by a system that is doing nothing, it's not an assertion, it's decoration.

Rule two: expected-empty is not unexpected-empty

Here's where I nearly overcorrected, and it's the subtlest part.

Some of my jobs legitimately do nothing most of the time. A watcher that looks for new items in a feed should find zero items on a quiet day. If I make "zero effects" a hard failure everywhere, those jobs scream every night and I'm back to a muted alert channel — which I already learned is worse than no alerts.

So the assertion can't be "produced > 0". It has to be "the source was successfully queried and the emptiness is trustworthy." That's a different check: the request succeeded, returned the expected shape, and the emptiness was an explicit, well-formed empty rather than a default. In my case the giveaway was available the whole time: the response had no pagination envelope. A real empty result would have come back with a cursor and a total. A silently-ignored-parameter empty came back as a bare [].

Practical form: for jobs that may legitimately produce nothing, assert on the envelope, not the count.

data = api.get("/filings", params={"from_date": since})
if not isinstance(data, dict) or "items" not in data:
    raise UnexpectedShapeError(f"got {type(data)}: {str(data)[:80]}")
items = data["items"]  # zero here is fine — we proved the query was honored

Now "quiet day" and "the API stopped honoring my request" look different, which is the entire job.

Rule three: three states, not two

This is what finally made it usable for both kinds of jobs. Every run now ends in one of three states, and they are not the same thing:

• succeeded_with_effect — it did the thing, with a count attached.
• no_work_needed — it did nothing and proved that nothing was the correct answer (envelope check above passed).
• failed — including JobProducedNothingError, i.e. it ran, and the absence of effect was not explained.

Only the third one alerts. The second one is a normal, boring, expected outcome that shows up in the briefing as a single quiet line. Before this split, "did nothing" was indistinguishable from "worked" — and the alerting I did have was tuned to a world where those were the same.

The part that actually caught things

Beyond the original bug, the effect counters caught two more silent failures in the next month: a token that expired and got silently replaced with an anonymous, lower-privilege session (200s, empty results), and a path change that made a downstream job read an empty directory and write a valid, empty file. Neither threw. Both were invisible to status-based monitoring. Both showed up instantly as a flatline in effects.

The dashboard line I now look at first isn't uptime or error rate. It's effects per run, per job. A job that runs and produces nothing, forever, is not a healthy job — it's a green light with the bulb unscrewed.

The transferable bit

If you take one thing: "no error" is not "worked." Your scheduler, your orchestrator, and your CI will happily report success for a process that queried nothing, wrote nothing, and sent nothing, because that is exactly what they were asked to measure.

Whatever your stack is — a nightly ETL, a webhook consumer, a CI step, an agent that posts things — pick one number per unit of work that would be zero if the work silently stopped. Assert on it. Track it over time. And make sure that some of your alerts can fire on a flatline, not just a spike. Spikes are loud and self-reporting. Flatlines need someone to be watching for them on purpose.

Curious how other people handle this: do you assert on effect counts in production jobs, or do you rely on downstream freshness checks (data arriving late → alert)? Both seem to have holes. Drop a comment with what's caught silent failures for you.

Part of my Building in Public series — previously: the guardrail stack I built before going live, timeout means no on approval gates, a circuit breaker that caught three outages, and a health monitor that tells me when my agents are dying.