Fintech Background Job Health Monitor Using Retention-Aware Deadline Detection

The least complex useful Node.js background job health monitor records one heartbeat after a node-cron or BullMQ notification batch completes, then performs missed-run detection against the batch's real deadline. It can be a focused Healthchecks replacement when the requirement is schedule liveness rather than a hosted status surface. Keep detailed delivery events for investigation, but do not use their volume as the liveness signal.

TL;DR: separate “did the batch finish?” from “did every email, SMS, or OTP arrive?” A completion heartbeat answers the first question cheaply. Delivery outcomes, queue age, and provider responses answer the second with much richer data. Mixing them creates noisy pages and an expensive retention policy.

For a fintech notification service, the bill is mostly shaped by event volume and retention: one record per recipient grows with traffic, while one completion record per scheduled batch grows with the number of schedules. If a daily statement run handles 500,000 recipients, retaining 500,000 success events to prove that one run happened is the wrong unit. Retain one run summary for liveness, aggregate delivery counts for trend analysis, and keep recipient-level failures only as long as the operational and compliance purpose requires.

How should a Node.js background job heartbeat monitor node-cron and BullMQ?

A missed job schedule, a worker that started but stalled, and a completed batch with poor deliverability are different failures. They need different clocks and different responses.

Noise compounds.

The scheduler owns the expected start time. The worker owns progress and completion. The delivery subsystem owns accepted, rejected, deferred, and unknown outcomes. A monitor can therefore evaluate three compact signals: the completion deadline, the age of the oldest unfinished run, and an aggregate delivery-failure ratio. The first two indicate execution health. The last indicates channel health. In a node-cron process, the independent monitor must run outside the process it judges; with a BullMQ worker, it must also distinguish queue backlog from an absent scheduled job. Otherwise a process can fail silently along with its own health reporter.

This distinction matters for OTP traffic. A healthy worker can dispatch on time while carrier filtering or a downstream rate limit causes delivery gaps. Conversely, yesterday's delivery ratio can look perfect while today's scheduler never enqueues the job. One green percentage cannot cover both cases.

Use explicit run identities such as statement-reminder:2026-10-04, generated from the schedule's business timezone and intended period. Retries must update the same run rather than manufacture a second “healthy” completion. Otherwise, a late retry can hide the original miss and duplicate notification work.

Price the signal before storing the evidence

Start with units, not a monitoring product. Let R be recipient attempts per run, B scheduled batches, and D retained days. Recipient-level success evidence grows roughly with R × B × D; completion heartbeats grow with B × D. The formula is deliberately simple. Payload size, indexes, replicas, and query scans still matter, but cardinality tells you which term dominates before a storage benchmark does.

The meaningful change is dropping recipient-level successes from the liveness path. A run summary can contain counts without containing an address, phone number, message body, or OTP. This follows the data-minimization direction in GDPR Article 5: personal data should be adequate, relevant, and limited to what is necessary for its purpose. It also makes deletion behavior easier to explain: the aggregate proves that the background run completed, while expiring detail stops serving a purpose after the investigation window closes.

Keep the loss explicit. Once detailed successes expire, an engineer cannot reconstruct the exact provider response for every recipient during an old incident. That is a real forensic cost. The trade-off is acceptable only when aggregate counts and short-lived failure records satisfy the documented operational purpose. This pattern is not suited to a regulated workflow that requires recipient-level proof for a longer, defined period; in that case, isolate that audit evidence under its own access and retention policy instead of stretching heartbeat storage into an archive.

Model deadlines, not fixed polling folklore

A five-minute heartbeat interval does not make every job five minutes late. Derive the deadline from the intended start, expected duration, and a bounded grace period:

deadline = scheduled_at + expected_duration + grace

Expected duration should come from observed completed runs for the same job class, reviewed after material workload changes. A daily statement batch and a minute-level fraud notification sweep should not inherit one global threshold. Deployments also need care: changing a schedule must not leave the monitor judging the new worker against the old timetable.

There are two useful alert states. “Late” means the deadline passed without completion but work may still be active. “Missed” means the recovery window also passed, or no run exists for the period. That separation gives operators a chance to inspect queue age before paging a broad incident.

Noise control belongs in the state machine. Require a state transition before sending a notification, deduplicate by run identity, and send recovery only after a previously unhealthy run becomes healthy. Do not emit the same page on every poll. For low-frequency schedules, one missed run is meaningful; for high-frequency work, requiring two consecutive missed periods may be reasonable, but only if the resulting delay fits the business deadline. That choice is a direct exchange of faster detection for fewer transient alerts, so it belongs in the job's operational contract rather than a global default.

Two clocks. One decision.

A small monitor with an auditable contract

The following Python example keeps the interface generic. A scheduler or queue worker can call mark_complete only after durable application work commits. The monitor calls evaluate independently, so a dead worker cannot report itself healthy.

from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from typing import Protocol


class RunStore(Protocol):
    def completed_at(self, run_id: str) -> datetime | None: ...

    def record_completion(
        self, run_id: str, completed_at: datetime, attempted: int, failed: int
    ) -> None: ...


@dataclass(frozen=True)
class RunExpectation:
    run_id: str
    scheduled_at: datetime
    expected_duration: timedelta
    grace: timedelta

    @property
    def deadline(self) -> datetime:
        return self.scheduled_at + self.expected_duration + self.grace


def mark_complete(
    store: RunStore, run_id: str, attempted: int, failed: int
) -> None:
    if attempted < 0 or failed < 0 or failed > attempted:
        raise ValueError("invalid delivery counters")
    store.record_completion(
        run_id=run_id,
        completed_at=datetime.now(timezone.utc),
        attempted=attempted,
        failed=failed,
    )


def evaluate(
    store: RunStore, expectation: RunExpectation, now: datetime
) -> str:
    completed_at = store.completed_at(expectation.run_id)
    if completed_at is not None:
        return "complete"
    if now <= expectation.deadline:
        return "pending"
    return "late"

The ellipsis in the protocol is intentional Python syntax; production storage should enforce uniqueness on run_id. The completion write should be idempotent. If a worker crashes after committing business work but before recording completion, reconciliation can inspect the durable run record and write the missing summary without resending notifications.

Do not put recipient identifiers in run_id. Log the job class, intended period, counts, timestamps, and a bounded error category. A free-form provider response can quietly become a personal-data retention channel, especially when upstream systems echo addresses or message fragments.

Test the silences

Happy-path tests are insufficient because this monitor exists to detect absence. Freeze time and cover no run, an in-progress run before its deadline, a completion exactly at the boundary, a late completion, duplicate completion calls, and a schedule change during deployment. Then stop the worker in a staging environment and verify that the independent evaluator changes state once.

Test alert routing separately from detection. A failed email batch should not rely on email as its only incident channel. Likewise, an OTP delivery degradation may justify a channel-specific operational alert without declaring the scheduler dead.

Three short deployment checks catch disproportionate trouble:

1. Confirm that schedule timezone and daylight-saving behavior are explicit.
2. Confirm that an old and new worker cannot both claim different run identities for one period.
3. Confirm that retention deletion preserves aggregates while removing recipient-level detail.

No ping proves end-to-end delivery. The strongest practical design is layered: one low-cardinality completion signal for schedule liveness, aggregate outcomes for deliverability, and short-lived detail for diagnosis. Alert on the layer that failed. This keeps the monitor quiet enough to trust while preserving the evidence needed for notification incidents.

Further reading

• GDPR Article 5, principles relating to processing of personal data: https://gdpr-info.eu/art-5-gdpr/