Here is a failure I had not thought about until someone asked me the right question.
You write a PreToolUse hook. It reads the tool call, decides ...
For further actions, you may consider blocking this person and/or reporting abuse
Good to see the thread grow into a post. The canary is the piece we never got to in the comments.
On detecting a guard that's been dead for a month, I'd count rather than simulate. A canary proves the guard still refuses the one call you scripted. It can't see the guard quietly losing its grip on real traffic. And logging inside the hook isn't enough either: if the matcher stops matching, the hook is never called and logs nothing, which looks exactly like a quiet week. So compare what the hook says it saw with a record the hook doesn't produce. If the remote shows twelve pushes this week and the push guard saw zero, it's dead, and you found out without guessing what the failure would look like.
It's your redaction bug turned around: the check has to come from a source that doesn't share the guard's belief. The VCS, CI logs and the harness transcript qualify. The hook's own fixtures don't.
Semitexa's
ai:verifyfollows the same rule for its own empty runs: when no check matches the change, the verdict isnothing_to_verify, not a pass, because "nothing ran" and "everything passed" must never look the same.Is there anything in your harness today that would give you that independent count, or would you have to build it?
Counting rather than simulating is better than what I had, and the reason is the part I'd missed: a canary only proves the guard still refuses the call I thought to script.
To your question, it turns out I already have the independent record and hadn't noticed it. The harness writes a JSONL transcript per session, and every tool call lands in it with its name and input, written by the harness rather than by any hook. I counted one of mine just now: 507 Bash calls, 328 edits, 186 writes. Hook refusals show up in there too, so both halves of your comparison come from the one source that doesn't share the guard's belief.
What makes that better than instrumenting forward is that it's retrospective. There are 594 of those transcripts sitting on my disk. So I can ask today whether the pooler guard has seen a matching command in the last month and find out it died in September, instead of adding a counter and waiting a month to learn the same thing. The audit I thought I'd have to wait for is already in the logs.
And agreed on the empty-run verdict. "Nothing ran" and "everything passed" being indistinguishable is the whole post.
And thank you for the comment, this is the most insightful comment I've had on this site.
Thank you, that means a lot from the person who wrote the post. The transcripts are a better answer than mine: written by the harness, retrospective, and already on disk, so the audit costs one script instead of a month of waiting. I'm curious what the count says for your pooler guard. And your thread already changed something on our side: today I filed a task to move our ai:verify gate out of the instructions file and into a Stop hook, with your "fail closed only where declining costs a prompt" rule and version-stamped recorded fixtures written into it.
I ran it. 599 transcripts, 81,200 Bash calls since July, 1,380 of them touching a pooler or RDS host, which is the traffic that guard claims to cover. 170 in the last full week, 82 this week, so the denominator is alive and current.
Then the useful part: I have a denominator and no numerator. The hook refuses things without recording that it refused them, so I can't complete your comparison today. The audit I was pleased with measures the half I already trusted.
One thing the series taught me that I'd have got wrong on my own. Two weeks in October show zero matching commands, which is exactly the shape I'd have read as a dead guard. They're zero because total Bash calls those weeks were also zero, since I wasn't working. So a per-hook count has to be read against total tool traffic, or a holiday and a dead tripwire are the same picture. Same mistake as the post, one level up, in the detector I built to catch it.
Good luck with the Stop hook. I'd be interested to hear whether fail-closed holds up once it's someone else's CI eating the false block. Hope you write a post about it.
Logging the raw input works better if nobody has to go read it. Store only the shape, something like
jq -c '[paths | map(strings) | join(".")] | unique'over each payload, hash it, and alert the first time a new hash shows up. A renamed field then makes noise on the day it happens instead of eighteen weeks later. It's the same trick that catches an error tracker going quiet: a sudden drop to zero events from a service usually means the SDK stopped sending, not that the bugs stopped. Does the harness put a version string in the payload you could key on too?Yeah, alerting on a new hash is better than what I wrote. Mine was "log it and read it sometimes", which means never.
One thing I'd change: a path-set hash moves on optional fields too, so the first Bash call that happens to include run_in_background gives you a new hash for no reason. But a rename shows up as a path vanishing and a path appearing, where an additive change only shows the appearance. The vanishing is what kills the matcher. So page on paths that stop showing up, just log the new ones.
On the version, no, it's not in the payload. I went and checked: session_id, prompt_id, transcript_path, cwd, scratchpad_dir, permission_mode, effort, hook_event_name, tool_name, tool_input, tool_use_id, plus agent_id and agent_type if a subagent made the call. No version anywhere.
transcript_path is in there though, and the transcript carries a version on every line. One of mine spans eight of them, 2.1.220 through 2.1.287, over about three months, because the CLI updates itself between sessions. So it's per call rather than per session, which is more useful than keying on it would have been. A new shape hash lands on a specific build instead of a date range.
Also tool_use_id, which I'd completely skipped over. It's in the payload and it's the same id the harness records against the call in its own log. So the numerator I was complaining about not having is one line in the hook, log the id and nothing else, and the join gets me the rest from the side that isn't the hook. Wouldn't have spotted that if you hadn't asked what was in the envelope.
The 'two states that look identical from the outside' line is the whole problem — a quiet guard reads as a working guard, and the broken case is worse than no guard because you'd have been more careful without one. Your warn-mode detail is the one I'd steal: a warning that can't name the file or command but talks as if it can teaches the model to discount every future warning. On the canary — have you tried making the scheduled bad call part of the hook's contract test, or does that just move the staleness problem one layer up?
It moves it, and I think I can say where to.
A contract test runs the hook against a payload the test runner hands it. If the harness renames a field, your known-bad call is still in the old shape, so it passes and proves the matcher works against last year's reality. You've tested your code rather than the integration, which is the original failure with a more respectable-looking test wrapped around it.
The canary only escapes that if the bad call goes through the real path: the agent issues it in a real session, the harness builds the payload, the hook refuses it where it normally would. Then the thing under test is the join, which is the part that actually broke.
So contract test for the matching logic, canary for the integration, and they go quiet in different ways. The warn-mode detail you mention I only got right because someone in the earlier thread pushed on it.
That split is the clean answer — contract test proves the matcher, canary proves the join, and neither can cover for the other going quiet. The 'different ways they go quiet' point is the one to watch: when the canary goes quiet, is that itself alarmed, or does it fail the same silent way the original guard did?
It fails the same silent way, unless you build it as the one kind of thing whose silence is already an alarm.
A guard is quiet by default, so nothing is expected and nothing notices. A scheduled canary is different: it is supposed to emit on a known cadence, so you can alert on the absence of a heartbeat instead of the presence of a failure. A dead man's switch rather than a check. Miss the daily ping and something pages, whether the canary crashed, the scheduler died or the host went away.
That is where the regress stops. You don't need a canary for the canary, you need the canary to be the layer at which silence is itself anomalous.
One condition though: the heartbeat has to be received by something outside your own setup, because the rule this whole thread keeps landing on is that the check can't share a belief with the thing it checks. An external pinger qualifies. A log file on the same box does not, which is Jackson's missing-log-file gate wearing a different hat.
The dead-man's-switch framing closes the regress neatly — silence is only anomalous when the heartbeat is expected, and expecting it from outside means the check finally stops sharing a fate with the thing it checks. The one detail I'd still pin down: does the heartbeat need to prove the canary actually ran and refused, or is a received ping on cadence enough — could a stale or replayed ping fake liveness?
A ping on cadence isn't enough, and it fails in the way this thread keeps rediscovering.
A bare heartbeat proves something on that box ran curl at the scheduled time. It says nothing about whether the canary issued the bad call, or whether the hook refused it. The classic version is a cron that runs the canary and then pings unconditionally, so it goes on pinging cheerfully for months after the canary starts exiting non-zero. That is Jackson's missing-log-file gate again, this time wearing a heartbeat.
So the ping has to be unreachable unless the assertion held. Last line of the script, exit on any failure, ping only when the hook actually refused. Then absence covers both "the canary never ran" and "the guard didn't refuse", two failure modes collapsing into one signal that can't be mistaken for health.
On replay: don't send a bare GET. Send the run timestamp and a hash of the refusal, so what gets attested is "the hook refused with the reason I expect" rather than "a packet arrived".
The honest limit is that none of this survives a machine that lies to you on purpose. But the adversary here isn't malice, it's a cron that outlives the thing it wraps. Against that one, pinging only on success is enough.
"A check built from the same assumption as the thing it checks is a second copy of your belief" is the line I will be repeating. The test and the hook agreed with each other perfectly and neither had spoken to the harness in months.
What I liked most is how the thread moved from simulating to counting. A canary proves the guard still refuses the one call you scripted. Counting real matches against the harness transcripts tells you whether it still sees live traffic, and running that per week, as Kartik suggested, dates the break instead of just reporting it.
Jackson's gate that passed when its log file was missing is the same bug in different clothes. A zero only means something next to a non-zero that proves the instrument was alive.
At Ascendra Ventures we have started writing "how do we know this guard is still working" into the handover for client automation, because the people who inherit it rarely know to ask. Now that you have a denominator, what would you accept as a healthy numerator for the pooler guard?
Good question, and trying to answer it made me notice I had picked the wrong numerator.
The instinct is to count blocks. That is wrong for this guard. Most of those 1,380 commands are perfectly legitimate psql against a pooler, and the bad shape is rare, so a healthy guard here might block nothing for months. Which puts me straight back inside the two-states-look-identical problem I wrote the post about.
The numerator that works is evaluations, not refusals. Every time the hook runs and reaches a decision, pass or block, that is a heartbeat, and a pass is evidence of life exactly as much as a block is. Most people log only refusals, which is precisely why the healthy state leaves no trace and the comparison can't be made. I log neither today, which is the real gap.
So the number I would accept: over any rolling 14 days containing at least 20 matching commands, evaluations should be at least 95 percent of them. The floor of 20 is what stops a holiday from paging. My own weekly series runs from 6 to 170, and any fortnight of actual work clears 20 comfortably.
And writing "how do we know this guard is still working" into the handover is a better idea than anything in the post. People who inherit a guard inherit its silence without inheriting any reason to be suspicious of it.
Same as a smoke detector nobody tests. It stays quiet for months and everyone reads quiet as safe. Pushing the button is the only way to know. We're building that kind of real-world check into ML Systems.
The analogy has a sharp edge that favours your point. The test button checks the siren, not the sensor. You press it, it shrieks, and you've learned the speaker works and nothing whatsoever about whether it can still smell smoke.
Which is the limit of a canary too: it proves the guard still refuses the one call you scripted. The sensor equivalent is counting real traffic against a record the guard doesn't produce.
The fix that keeps working after the harness changes again: stop building the payload in the test by hand. Record it. Capture the last dozen real PreToolUse calls your harness actually makes — they're small, and nothing sensitive lives in the shape — commit them as golden files, and have the unit test assert the hook matches at least one recent real payload.
Then a harness field rename turns your CI red within a day instead of leaving the tripwire dead for eighteen weeks. The test stops being a mirror of your belief and becomes a diff against the outside world, which is the thing that actually changed.
This composes with the canary idea rather than replacing it: the canary proves the hook still fires at runtime, golden payloads prove the matching logic still describes reality. You want both, because they go quiet in different ways.
Agreed, and "at least one recent real payload" is carrying more weight in that sentence than it looks.
A corpus recorded once and committed becomes the same stale belief with better provenance. Two years of green tests against a dozen payloads from setup day proves nothing about today. So the assertion has to be against payloads newer than some window, which means the recording can't be a one-off, it has to keep running.
Someone in the other thread added the piece that makes it enforceable: stamp each recording with the harness version, and have CI refuse a changed fixture whose stamp didn't move, because re-recording is the reflex that turns a red test green without anyone deciding anything.
And yes on composing rather than replacing. Golden payloads check that the matcher still describes reality, the canary checks the integration still fires. I have neither, so at the moment I'm two good ideas and no implementation.
One of my gates passed whenever its log file was missing. It enforced the hardest rule in my agent test harness, no live API calls outside the planned captures, by reading the call log. If the file didn't exist it returned True with "absent, so zero calls logged", so a recorder that wrote nothing would have passed. An agent found it while reading the gate code. The fix makes a zero count only when a non-zero next to it shows the recorder ran.
That's the same bug in different clothes, and your fix names the general form better than my post did.
A zero is ambiguous. It means "nothing happened" or "nothing was recorded", and from the inside those are identical. So a zero is only trustworthy next to a non-zero that proves the instrument was alive, which is exactly your check.
I hit it an hour ago counting how often one of my guards should have fired. Two weeks in October showed zero matching commands. Dead guard, obviously. They were zero because I hadn't run the agent at all those weeks, and the only reason I know is a second count sitting beside the first.
Also worth sitting with: your gate was found by an agent reading the code, and mine was found by a GIF I made for a landing page. Neither of those is a process.
A harness renamed one field and the tests stayed green because the test built its own payload. A check I'd add is a canary run: on a schedule, feed the hook a payload captured from the real harness rather than one written by hand, and require that a known-dangerous command is actually blocked. I'd also keep a counter of how often the hook matched anything during normal use and alert when it flatlines. That way a quiet hook and a broken hook stop looking identical, and the unit tests no longer have to carry that job alone.
Both halves of that are right, and the thread has turned up a limit on each that's worth knowing before you build them.
On the canary: if the scheduled bad call is handed to the hook by a test runner, it only proves the matcher works on the payload shape you recorded. To exercise the thing that actually breaks, the call has to go through the real harness so the payload gets built the way a real one is.
On the flatline alert: mine would have paged twice in October for a completely healthy hook, because I wasn't working those weeks and every count was zero. It needs a per-hook baseline and a denominator of total tool traffic rather than a global threshold.
@kartik-nvjk and @kami_butt_69b922ade8de012 in this comment section reached those 2 points independently, which probably means they're the real edges rather than my particular mess.
Since the transcripts are already on disk, I'd run the count per hook per week instead of one total. A hook whose matches drop to zero in a single week points straight at the harness update that renamed the field, which matters when the break is eighteen weeks old.
You're right, and I ran it per week within the hour. It dates the break, which a total can't: "it's dead" becomes "it died the week of the 12th", and now there's something specific to go and look at.
One wrinkle I only saw because the data showed it. Two weeks in my series are zero, which is the exact shape of a hook that stopped matching. They're zero because I wasn't working those weeks and total Bash calls were also zero. So the weekly series has to be read as a share of overall tool traffic rather than as a raw count, or a holiday and a dead tripwire look the same. Which is the post's own failure mode turning up one level up, in the detector I built to catch it.