Here is a failure I had not thought about until someone asked me the right question.
You write a PreToolUse hook. It reads the tool call, decides the command is dangerous, and blocks it. You write unit tests for it, because a guard with no tests is a wish. The tests pass. Months go by and the hook never fires, which you read as good news, because nothing bad has happened.
Then one day you find out the harness renamed a field in the payload eighteen weeks ago, your hook has matched nothing since, and every one of your tests is still green. They're green because the test builds the payload itself, in the shape you believed the harness sends. The test and the hook agree with each other perfectly. Neither of them has spoken to the harness in months.
Green tests, dead tripwire. The guard and its tests share a belief, so the tests cannot see the belief go stale.
This is the same shape as a bug I wrote about last time, where a redaction tool verified its own output using the same wrong geometry it had used to draw with. Different domain, identical structure: a check built from the same assumption as the thing it checks is not a check, it's a second copy of your belief.
Why "it hasn't fired" tells you nothing
A guard has two states that look identical from the outside. Working and quiet, because nothing tripped it. Broken and quiet, because it cannot match anything any more.
Nothing in your workflow distinguishes them. You don't get a weekly report of hooks that ran. The absence of refusals feels like safety, and in the broken case it is precisely the opposite: you've been running unguarded while believing you were covered, which is worse than having no hook at all, because you'd have been more careful without one.
Every guard that encodes a fact about an external world has this problem. The payload shape of your agent harness. The name of a tool. Whether CI still runs the check your hook assumes runs. Those facts change without telling you, and the guard goes quiet rather than loud.
Fail closed, but not everywhere
The obvious fix is to fail closed. If this is a tool I'm guarding and the field I need isn't there, that's an anomaly, not an allow. Most hooks, mine included, default to allowing anything they can't parse, which is exactly backwards.
The question I got asked, and did not have a good answer for at first, was whether that means failing closed everywhere, or only for guards on irreversible things.
I now think irreversibility is the wrong axis. The right one is what happens when the hook declines to decide. Three modes, three correct failures:
Block mode: fail closed. The hook is certain from the input alone that the call is wrong. If it can't read the input, refuse. The fallback is a retry or a question, both cheap, and the false block you eat on the day the harness changes is the thing that tells you the shape moved. That one annoying refusal is the monitoring you never set up.
Warn mode: don't fail closed, and say why. A warn hook exists because the rule is a heuristic that will fire on legitimate work, so turning an unparseable payload into a wall contradicts its own design. Emit the warning anyway. And the detail I'd have got wrong: the warning has to say precisely that it couldn't read the input, rather than printing its usual generic text. A warning that can't name the file or the command, but talks as if it can, teaches the model to discount every future warning from that hook.
Auto-allow mode: already fails closed, for free. This is the mode nobody discusses. Hooks that pre-approve read-only commands so they never surface a permission prompt. If one can't parse the input, it simply doesn't allow, and you land on the normal prompt. The fallback is a human, which is exactly where you want to be.
So the rule I'd write now: fail closed wherever declining costs a prompt. Don't, wherever declining costs a block on legitimate work.
The fixture that lies to you
The other half is contract tests, and there's a trap in them.
Recording one real payload from an actual session and replaying it against the hook catches the rename before it costs you anything. Writing that fixture by hand does not, because a hand-written fixture is your belief about the payload shape stored in a second file. You now have two copies of the same assumption and a false sense of coverage. If you can't capture a real one, log the raw input on every invocation and go read it occasionally. Dull, works.
My own setup, honestly: two hooks have real unit tests, one with a table of about thirty adversarial command variants, and both assert the deny path rather than only the happy one. Neither has a recorded fixture. I haven't fixed it yet.
What I still don't have
The thing I'd actually want is a canary. A scheduled job that deliberately issues a known-bad call and asserts it gets refused, so a dead guard announces itself instead of waiting to be discovered. That's a cron and an assertion, it isn't hard, and I haven't built it, which I suspect means most people running a dozen hooks haven't either.
Most of this post came out of the comments on my last one rather than from me. @build996 pushed on which refusals an agent can actually act on, @max_quimby on hooks quietly becoming an unversioned rulebook nobody remembers the reasons for, @syntaxwanderer_26 asked the question that produced the whole fail-closed taxonomy and then supplied the warn-mode detail, and @doriansotpyrc asked how a rule earns promotion from prose into a hook, which I answered as: once means write the sentence, twice means the sentence isn't carrying it.
So, the part I'm stuck on. How would you detect a guard that has been dead for a month? A canary is the only honest answer I've got, and it only covers the failures you thought to simulate. If you've found something better, or you've had a tripwire die quietly and only noticed by accident, I'd like to hear it.
Top comments (32)
Good to see the thread grow into a post. The canary is the piece we never got to in the comments.
On detecting a guard that's been dead for a month, I'd count rather than simulate. A canary proves the guard still refuses the one call you scripted. It can't see the guard quietly losing its grip on real traffic. And logging inside the hook isn't enough either: if the matcher stops matching, the hook is never called and logs nothing, which looks exactly like a quiet week. So compare what the hook says it saw with a record the hook doesn't produce. If the remote shows twelve pushes this week and the push guard saw zero, it's dead, and you found out without guessing what the failure would look like.
It's your redaction bug turned around: the check has to come from a source that doesn't share the guard's belief. The VCS, CI logs and the harness transcript qualify. The hook's own fixtures don't.
Semitexa's
ai:verifyfollows the same rule for its own empty runs: when no check matches the change, the verdict isnothing_to_verify, not a pass, because "nothing ran" and "everything passed" must never look the same.Is there anything in your harness today that would give you that independent count, or would you have to build it?
Counting rather than simulating is better than what I had, and the reason is the part I'd missed: a canary only proves the guard still refuses the call I thought to script.
To your question, it turns out I already have the independent record and hadn't noticed it. The harness writes a JSONL transcript per session, and every tool call lands in it with its name and input, written by the harness rather than by any hook. I counted one of mine just now: 507 Bash calls, 328 edits, 186 writes. Hook refusals show up in there too, so both halves of your comparison come from the one source that doesn't share the guard's belief.
What makes that better than instrumenting forward is that it's retrospective. There are 594 of those transcripts sitting on my disk. So I can ask today whether the pooler guard has seen a matching command in the last month and find out it died in September, instead of adding a counter and waiting a month to learn the same thing. The audit I thought I'd have to wait for is already in the logs.
And agreed on the empty-run verdict. "Nothing ran" and "everything passed" being indistinguishable is the whole post.
And thank you for the comment, this is the most insightful comment I've had on this site.
Thank you, that means a lot from the person who wrote the post. The transcripts are a better answer than mine: written by the harness, retrospective, and already on disk, so the audit costs one script instead of a month of waiting. I'm curious what the count says for your pooler guard. And your thread already changed something on our side: today I filed a task to move our ai:verify gate out of the instructions file and into a Stop hook, with your "fail closed only where declining costs a prompt" rule and version-stamped recorded fixtures written into it.
I ran it. 599 transcripts, 81,200 Bash calls since July, 1,380 of them touching a pooler or RDS host, which is the traffic that guard claims to cover. 170 in the last full week, 82 this week, so the denominator is alive and current.
Then the useful part: I have a denominator and no numerator. The hook refuses things without recording that it refused them, so I can't complete your comparison today. The audit I was pleased with measures the half I already trusted.
One thing the series taught me that I'd have got wrong on my own. Two weeks in October show zero matching commands, which is exactly the shape I'd have read as a dead guard. They're zero because total Bash calls those weeks were also zero, since I wasn't working. So a per-hook count has to be read against total tool traffic, or a holiday and a dead tripwire are the same picture. Same mistake as the post, one level up, in the detector I built to catch it.
Good luck with the Stop hook. I'd be interested to hear whether fail-closed holds up once it's someone else's CI eating the false block. Hope you write a post about it.
Logging the raw input works better if nobody has to go read it. Store only the shape, something like
jq -c '[paths | map(strings) | join(".")] | unique'over each payload, hash it, and alert the first time a new hash shows up. A renamed field then makes noise on the day it happens instead of eighteen weeks later. It's the same trick that catches an error tracker going quiet: a sudden drop to zero events from a service usually means the SDK stopped sending, not that the bugs stopped. Does the harness put a version string in the payload you could key on too?Yeah, alerting on a new hash is better than what I wrote. Mine was "log it and read it sometimes", which means never.
One thing I'd change: a path-set hash moves on optional fields too, so the first Bash call that happens to include run_in_background gives you a new hash for no reason. But a rename shows up as a path vanishing and a path appearing, where an additive change only shows the appearance. The vanishing is what kills the matcher. So page on paths that stop showing up, just log the new ones.
On the version, no, it's not in the payload. I went and checked: session_id, prompt_id, transcript_path, cwd, scratchpad_dir, permission_mode, effort, hook_event_name, tool_name, tool_input, tool_use_id, plus agent_id and agent_type if a subagent made the call. No version anywhere.
transcript_path is in there though, and the transcript carries a version on every line. One of mine spans eight of them, 2.1.220 through 2.1.287, over about three months, because the CLI updates itself between sessions. So it's per call rather than per session, which is more useful than keying on it would have been. A new shape hash lands on a specific build instead of a date range.
Also tool_use_id, which I'd completely skipped over. It's in the payload and it's the same id the harness records against the call in its own log. So the numerator I was complaining about not having is one line in the hook, log the id and nothing else, and the join gets me the rest from the side that isn't the hook. Wouldn't have spotted that if you hadn't asked what was in the envelope.
The 'two states that look identical from the outside' line is the whole problem — a quiet guard reads as a working guard, and the broken case is worse than no guard because you'd have been more careful without one. Your warn-mode detail is the one I'd steal: a warning that can't name the file or command but talks as if it can teaches the model to discount every future warning. On the canary — have you tried making the scheduled bad call part of the hook's contract test, or does that just move the staleness problem one layer up?
It moves it, and I think I can say where to.
A contract test runs the hook against a payload the test runner hands it. If the harness renames a field, your known-bad call is still in the old shape, so it passes and proves the matcher works against last year's reality. You've tested your code rather than the integration, which is the original failure with a more respectable-looking test wrapped around it.
The canary only escapes that if the bad call goes through the real path: the agent issues it in a real session, the harness builds the payload, the hook refuses it where it normally would. Then the thing under test is the join, which is the part that actually broke.
So contract test for the matching logic, canary for the integration, and they go quiet in different ways. The warn-mode detail you mention I only got right because someone in the earlier thread pushed on it.
That split is the clean answer — contract test proves the matcher, canary proves the join, and neither can cover for the other going quiet. The 'different ways they go quiet' point is the one to watch: when the canary goes quiet, is that itself alarmed, or does it fail the same silent way the original guard did?
It fails the same silent way, unless you build it as the one kind of thing whose silence is already an alarm.
A guard is quiet by default, so nothing is expected and nothing notices. A scheduled canary is different: it is supposed to emit on a known cadence, so you can alert on the absence of a heartbeat instead of the presence of a failure. A dead man's switch rather than a check. Miss the daily ping and something pages, whether the canary crashed, the scheduler died or the host went away.
That is where the regress stops. You don't need a canary for the canary, you need the canary to be the layer at which silence is itself anomalous.
One condition though: the heartbeat has to be received by something outside your own setup, because the rule this whole thread keeps landing on is that the check can't share a belief with the thing it checks. An external pinger qualifies. A log file on the same box does not, which is Jackson's missing-log-file gate wearing a different hat.
The dead-man's-switch framing closes the regress neatly — silence is only anomalous when the heartbeat is expected, and expecting it from outside means the check finally stops sharing a fate with the thing it checks. The one detail I'd still pin down: does the heartbeat need to prove the canary actually ran and refused, or is a received ping on cadence enough — could a stale or replayed ping fake liveness?
A ping on cadence isn't enough, and it fails in the way this thread keeps rediscovering.
A bare heartbeat proves something on that box ran curl at the scheduled time. It says nothing about whether the canary issued the bad call, or whether the hook refused it. The classic version is a cron that runs the canary and then pings unconditionally, so it goes on pinging cheerfully for months after the canary starts exiting non-zero. That is Jackson's missing-log-file gate again, this time wearing a heartbeat.
So the ping has to be unreachable unless the assertion held. Last line of the script, exit on any failure, ping only when the hook actually refused. Then absence covers both "the canary never ran" and "the guard didn't refuse", two failure modes collapsing into one signal that can't be mistaken for health.
On replay: don't send a bare GET. Send the run timestamp and a hash of the refusal, so what gets attested is "the hook refused with the reason I expect" rather than "a packet arrived".
The honest limit is that none of this survives a machine that lies to you on purpose. But the adversary here isn't malice, it's a cron that outlives the thing it wraps. Against that one, pinging only on success is enough.
Ping-only-on-success is the clean close to my question — it collapses 'never ran' and 'didn't refuse' into one absence signal, and the timestamp plus refusal hash turns the ping from mere liveness into attestation of the exact refusal. The scope limit is the honest part too: it defends against drift and a cron outliving its guard, not against a machine lying on purpose, and those are different adversaries. The one detail I'd still want pinned down: does the hash cover the full refusal payload, or just the reason code — in other words, how much of the refusal would have to change before the attestation stops matching?
Closer to the reason code than the full payload, but with the decision value in there, which I think is the part that matters.
Hashing the whole refusal doesn't work. My reason string is free text meant for the model to read and it quotes the offending command back, so the hash changes every run. I'd be updating the expected value constantly, which is the same re-recording habit that makes the fixtures worthless.
But the reason code alone is too loose. The decision is an enum (allow, deny, ask, defer) and a guard that quietly went from deny to ask would still read as a refusal, with the same reason code on it. So the decision needs to be in the hash and the prose doesn't.
So: hook name, which check inside it fired, the decision, the harness version. The version's in there deliberately, so an upgrade breaks the attestation and I have to go and look. Same idea as the version-stamped fixtures earlier in the thread.
Though three of those four fields are things I made up, so it's my schema again. The decision is the only one that comes back from the harness, and it's the field doing the actual work.
Putting the decision in the hash is the load-bearing choice — a guard quietly drifting from deny to ask would otherwise still read as a refusal, and the version stamp forcing a re-look on upgrade is the same discipline as the version-stamped fixtures. The honest limit is the one you name: hook name and check are your schema, not harness returns, so the attestation proves the record is intact, not that the hook itself is honest — is that a limit you accept, or does the harness need to return the check identity too?
I was going to say I accept the limit, then I went and looked, and the harness does return it. I had this wrong.
I'd been searching the transcripts for hook-shaped records. Those exist, 126,219 of them, one per hook invocation with the hook's stdout captured verbatim, and every single one is an exit 0. Not one refusal in the lot, which is why I told you I had a denominator and no numerator.
The refusals are there, in a different shape. A blocked call goes back to the model as a tool result, and the harness writes that result into the transcript like any other. It reads PreToolUse:Bash hook error: [python3 -S .../prod-db-guard.py]: Blocked: ... with the reason attached. Event, tool, the literal hook command, the full reason, keyed to the tool_use id, none of it written by the hook. I was looking in the wrong half of my own log.
Which answers your question better than my guess would have. Hook identity comes back from the harness, so that field doesn't belong in my schema at all. What stays self-reported is only which branch inside the hook fired, since that lives in the reason string the hook composes. I think that's a contract test's job rather than an attestation's, and making the attestation cover it is asking one instrument to do two jobs.
Two things if you build this. Some of mine are recorded as [command hook 16] instead of a path, because the hook was registered inline, so the identity is positional and reordering the settings file silently renames it in the historical record. And the numbers: 99 refusals total since July, of which the pooler guard is 2, against the 1,380 commands I measured it covering. The guard I was proudest of is the one that has barely fired.
That correction is worth more than the original answer — the refusals were in the transcript all along, just in tool-result shape rather than hook-record shape, so the harness does return the identity after all. The positional-identity wrinkle is the one I'd watch: if reordering the settings file silently renames [command hook 16], the historical record stops being comparable. And the 2-of-1380 number is a useful humility check — is the pooler guard barely firing because the covered commands are rarely dangerous, or because the danger mostly lives elsewhere?
Neither, as it turns out, and your question splits the guard in half. I classified the traffic instead of guessing.
Of the pooler-touching commands, 983 are plain SELECTs, which the guard ignores on purpose. Zero contained a destruction verb. No TRUNCATE, no DROP TABLE, no DELETE without a WHERE, no FLUSHALL, in four months. So that half isn't barely firing, it has never once been eligible to fire. Its zero is correct and uninformative, and it always will be, because it's a backstop sitting behind another check that runs first. A backstop refusing regularly would mean the thing in front of it had failed.
The other half is the lead. 75 of those commands look like a session-level SET heading for a prod host, which is precisely the shape that guard exists for, and it refused 2. I can't explain that gap yet. Maybe the guard is just narrower than my classifier, since its matcher wants the SET reaching psql in a specific form and I wrote mine in two minutes. Maybe it stopped matching some of them.
Either way it's answerable without building anything, and it's the golden-payload idea from upthread arriving for free. I have 75 real commands from real sessions already sitting in the transcripts. Replay them through the current hook and count how many it claims. Refuse all 75 and the guard is healthy and my classifier is wrong. Refuse 2 and I have a dead branch plus 73 commands that walked straight through. No fixture to author, because the harness recorded the corpus for me.
So the honest version of my earlier answer: "barely firing" was me reading one number against a denominator that didn't match it. One half has no eligible traffic and never did. The other half has 75 candidates and 2 refusals, and the replay will say which story is true.
That's a much better answer than the number alone — the replay settles it without building anything, and the two outcomes are cleanly different. If it refuses only 2 of the 75, the interesting question becomes whether the guard is narrower than it should be or the matcher has quietly stopped matching: would you widen the guard to the classifier's shape, or treat the mismatch itself as the bug to chase first?
"A check built from the same assumption as the thing it checks is a second copy of your belief" is the line I will be repeating. The test and the hook agreed with each other perfectly and neither had spoken to the harness in months.
What I liked most is how the thread moved from simulating to counting. A canary proves the guard still refuses the one call you scripted. Counting real matches against the harness transcripts tells you whether it still sees live traffic, and running that per week, as Kartik suggested, dates the break instead of just reporting it.
Jackson's gate that passed when its log file was missing is the same bug in different clothes. A zero only means something next to a non-zero that proves the instrument was alive.
At Ascendra Ventures we have started writing "how do we know this guard is still working" into the handover for client automation, because the people who inherit it rarely know to ask. Now that you have a denominator, what would you accept as a healthy numerator for the pooler guard?
Good question, and trying to answer it made me notice I had picked the wrong numerator.
The instinct is to count blocks. That is wrong for this guard. Most of those 1,380 commands are perfectly legitimate psql against a pooler, and the bad shape is rare, so a healthy guard here might block nothing for months. Which puts me straight back inside the two-states-look-identical problem I wrote the post about.
The numerator that works is evaluations, not refusals. Every time the hook runs and reaches a decision, pass or block, that is a heartbeat, and a pass is evidence of life exactly as much as a block is. Most people log only refusals, which is precisely why the healthy state leaves no trace and the comparison can't be made. I log neither today, which is the real gap.
So the number I would accept: over any rolling 14 days containing at least 20 matching commands, evaluations should be at least 95 percent of them. The floor of 20 is what stops a holiday from paging. My own weekly series runs from 6 to 170, and any fortnight of actual work clears 20 comfortably.
And writing "how do we know this guard is still working" into the handover is a better idea than anything in the post. People who inherit a guard inherit its silence without inheriting any reason to be suspicious of it.
Same as a smoke detector nobody tests. It stays quiet for months and everyone reads quiet as safe. Pushing the button is the only way to know. We're building that kind of real-world check into ML Systems.
The analogy has a sharp edge that favours your point. The test button checks the siren, not the sensor. You press it, it shrieks, and you've learned the speaker works and nothing whatsoever about whether it can still smell smoke.
Which is the limit of a canary too: it proves the guard still refuses the one call you scripted. The sensor equivalent is counting real traffic against a record the guard doesn't produce.
The fix that keeps working after the harness changes again: stop building the payload in the test by hand. Record it. Capture the last dozen real PreToolUse calls your harness actually makes — they're small, and nothing sensitive lives in the shape — commit them as golden files, and have the unit test assert the hook matches at least one recent real payload.
Then a harness field rename turns your CI red within a day instead of leaving the tripwire dead for eighteen weeks. The test stops being a mirror of your belief and becomes a diff against the outside world, which is the thing that actually changed.
This composes with the canary idea rather than replacing it: the canary proves the hook still fires at runtime, golden payloads prove the matching logic still describes reality. You want both, because they go quiet in different ways.
Agreed, and "at least one recent real payload" is carrying more weight in that sentence than it looks.
A corpus recorded once and committed becomes the same stale belief with better provenance. Two years of green tests against a dozen payloads from setup day proves nothing about today. So the assertion has to be against payloads newer than some window, which means the recording can't be a one-off, it has to keep running.
Someone in the other thread added the piece that makes it enforceable: stamp each recording with the harness version, and have CI refuse a changed fixture whose stamp didn't move, because re-recording is the reflex that turns a red test green without anyone deciding anything.
And yes on composing rather than replacing. Golden payloads check that the matcher still describes reality, the canary checks the integration still fires. I have neither, so at the moment I'm two good ideas and no implementation.
One of my gates passed whenever its log file was missing. It enforced the hardest rule in my agent test harness, no live API calls outside the planned captures, by reading the call log. If the file didn't exist it returned True with "absent, so zero calls logged", so a recorder that wrote nothing would have passed. An agent found it while reading the gate code. The fix makes a zero count only when a non-zero next to it shows the recorder ran.
That's the same bug in different clothes, and your fix names the general form better than my post did.
A zero is ambiguous. It means "nothing happened" or "nothing was recorded", and from the inside those are identical. So a zero is only trustworthy next to a non-zero that proves the instrument was alive, which is exactly your check.
I hit it an hour ago counting how often one of my guards should have fired. Two weeks in October showed zero matching commands. Dead guard, obviously. They were zero because I hadn't run the agent at all those weeks, and the only reason I know is a second count sitting beside the first.
Also worth sitting with: your gate was found by an agent reading the code, and mine was found by a GIF I made for a landing page. Neither of those is a process.
A harness renamed one field and the tests stayed green because the test built its own payload. A check I'd add is a canary run: on a schedule, feed the hook a payload captured from the real harness rather than one written by hand, and require that a known-dangerous command is actually blocked. I'd also keep a counter of how often the hook matched anything during normal use and alert when it flatlines. That way a quiet hook and a broken hook stop looking identical, and the unit tests no longer have to carry that job alone.
Both halves of that are right, and the thread has turned up a limit on each that's worth knowing before you build them.
On the canary: if the scheduled bad call is handed to the hook by a test runner, it only proves the matcher works on the payload shape you recorded. To exercise the thing that actually breaks, the call has to go through the real harness so the payload gets built the way a real one is.
On the flatline alert: mine would have paged twice in October for a completely healthy hook, because I wasn't working those weeks and every count was zero. It needs a per-hook baseline and a denominator of total tool traffic rather than a global threshold.
@kartik-nvjk and @kami_butt_69b922ade8de012 in this comment section reached those 2 points independently, which probably means they're the real edges rather than my particular mess.
Since the transcripts are already on disk, I'd run the count per hook per week instead of one total. A hook whose matches drop to zero in a single week points straight at the harness update that renamed the field, which matters when the break is eighteen weeks old.
You're right, and I ran it per week within the hour. It dates the break, which a total can't: "it's dead" becomes "it died the week of the 12th", and now there's something specific to go and look at.
One wrinkle I only saw because the data showed it. Two weeks in my series are zero, which is the exact shape of a hook that stopped matching. They're zero because I wasn't working those weeks and total Bash calls were also zero. So the weekly series has to be read as a share of overall tool traffic rather than as a raw count, or a holiday and a dead tripwire look the same. Which is the post's own failure mode turning up one level up, in the detector I built to catch it.