At 08:33 one morning I ran GET /api/sessions against my own WhatsApp gateway and read the line I least wanted to read:
{
"name": "session_<redacted>",
"status": "FAILED",
"me": { "pushName": "<business name>", "id": "<redacted>@c.us" },
"timestamps": { "activity": "<days ago>" }
}
The last activity on that session was days old. The line had been dead for days, and every dashboard I own was green.
I run WhatsApp automation for Israeli small businesses. Each client's business number lives as a session inside a self-hosted WAHA container, which bridges their line into a CRM and a pile of n8n workflows. Several businesses, one container.
Over a few days in September, several of those lines went down. Here is what they taught me, and what I check now.
The health check was answering the wrong question
My monitor checked that the waha container was running and responsive. It was. It stayed running, responsive and healthy through every one of those outages, because a container that is perfectly happy to serve you a list of dead sessions is, by every definition Docker cares about, working.
This is the ordinary liveness-versus-readiness mistake, except the readiness that mattered was not the service's. It was the readiness of the independent sessions inside the service, each of which can fail on its own, for unrelated reasons, without the process noticing or caring.
One client's line failed one evening and I found it days later; in between, the container reported itself healthy hundreds of times.
The failure that looks like nothing at all
This failure mode is the reason I am writing this.
The gateway logs a device removed stream error when the linked device is removed from the phone, either on purpose or while someone clears out old sessions. Which of the two it was hardly matters.
What matters is what the business sees afterwards: nothing. Messages keep arriving on the owner's phone exactly as before. Customers still get replies, because a human is reading the phone. From the business's side the line looks completely, visibly fine.
Only the CRM goes blind. Only the automations stop firing. The scheduled reminders, the routing, the logging, the follow-ups — all silently not happening, behind a line that looks perfectly alive to the only person in a position to notice.
If your failure mode is invisible to the user and invisible to your health check, you do not have a monitoring gap. You have no monitoring.
You cannot always tell whose line just died
When I sat down to write the alerting, I hit a problem I had not anticipated: a dead session may not be able to tell you who it belongs to.
While a session is WORKING, /api/sessions gives you me.id (the number) and me.pushName (the business name). When a session is torn down by an unlink, those fields are wiped. The identity disappears at precisely the moment you need it to address an alert.
I want to be careful here, because I wrote this down as an absolute rule in my own notes last week and it is not one. Checking the live API this morning while drafting this post, the currently-failed line does still carry its me block, as you can see in the JSON above. So it depends on how the session died: a removed device wipes the identity, a connection-level failure does not.
Which is worse than either rule on its own, because it means you cannot depend on the field being there. The fix is the same either way: keep your own roster. My watcher rewrites a roster.json on every pass — but only from sessions currently in WORKING, because a single pass during an outage would otherwise overwrite good identity with nulls.
And you cannot recover the mapping from billing, either. I searched our accounting system for these business numbers and found none: what sits in an invoice is the owner's personal mobile, which is a different number from the business line the bot runs on. The mapping is manual, or it does not exist.
The watchdog that never ran
On 8 September I built the thing that should have caught all of this: a watcher on a ten-minute cron that diffs session states, notifies the business owner when their own line drops, and sends them a pairing code to reconnect it themselves.
I tested it. Ten test scenarios, three real bugs caught before install. I watched it run. I marked the problem solved.
It did not run once from cron until 13 September.
*/10 * * * * sudo /usr/local/bin/waha-session-watch >> /opt/waha-watch/cron.log 2>&1
/opt/waha-watch is owned by root, mode 755. The cron user is waha. The redirect is evaluated by the shell before sudo runs, as the unprivileged user, so creating cron.log failed, so the shell exited, so the command never executed at all. Cron dutifully tried to mail me the error and found no MTA installed, so it dropped it:
(CRON) info (No MTA installed, discarding output)
Five days of no monitoring, hidden behind a crontab entry that reads perfectly.
What fooled me during the install was watch.log. It had runs in it — 21:40, 21:52, 21:59, 22:22. Non-round minutes, every one of them, because they were my own manual test runs. Scheduled runs land on round minutes. I looked at a log full of evidence that the tool worked and read it as evidence that the schedule worked.
Another line fell into that exact blind window. The first genuinely scheduled run found it immediately, days late.
The fix was touch cron.log && chown waha:waha. That is the entire fix. The lesson costs more than the fix: that No MTA installed, discarding output line in syslog is never noise. It means your scheduled command printed something nobody has ever read.
What I check now
Four things changed:
Health-check the units, not the process. The container being up says nothing about the sessions inside it. Whatever your service multiplexes — tenants, sessions, connections, device links — that is the thing with a state worth alerting on.
Verify the first scheduled run, in two places. Not "is it in crontab -l", which was true and meaningless for five days. Check syslog for the CMD line, and check the tool's own log for an entry on a round minute. If a cron entry redirects to a file, create the file first, owned by the cron user.
Keep a persistent roster, written only from healthy state. Never let a failure pass overwrite identity you will need to send the alert.
Alert the person who can actually fix it. A dropped line is repaired by someone holding the phone, not by me. The watcher now messages the business owner directly with a pairing code, inside working hours, once per incident. The one-message-per-incident rule matters as much as the alert: the fastest way to get a monitoring channel ignored is to send the same thing every ten minutes.
It matters that this is WhatsApp specifically. Bezeq's 2025 Internet Report — the annual survey of Israeli internet use, published in December 2025 — notes in passing that reaching 90% WhatsApp usage in Israel took about a decade. For a small business here, that line is not a channel. It is the front door. Which is exactly why the customer-service bots running on those lines need a liveness signal independent of the phone in the owner's pocket, because the owner's pocket will never report the outage. It does not look like one from there.
Several lines, a few days, one container that never once said anything was wrong.
The question I am actually asking
I want to hear about the second category specifically — not the outage your monitoring missed, but the one where the user experience stayed completely normal while the system underneath was dead, so nobody had any reason to report it.
What was the failure, and what signal finally caught it? I am collecting these, because I suspect the honest answer for most of them is "a human noticed weeks later," and I would like to be proven wrong.
Top comments (0)