Build step four of Sentinel is the freshness watchdog and derived points. It was audited about four weeks ago against the document that owns it, The Watchdog and Derived Points, which went out last Wednesday.
That post promised this entry would say where the specification and the build disagree, and which of the two had to change. For this step the answer is usually both. In one case the code changed, then the decision that explained the code, then the document, in that order and each because of the one before.
Half a sentence
The published copy says this about a stale point recovering:
An observation with
observed_atolder than the current deadline - a buffered replay landing late - does not clear staleness, though it is still logged.
There are two claims in it. A late observation must not push the point's deadline out, and it must not change the point's quality.
The first was built. Ingest hands the watchdog each observation's observed_at, so a replay of hours-old data schedules a deadline that is already in the past.
The second was not, and could not have been where the code stood. The projection applies an observation's quality without ever seeing a deadline; the information is not available to it. A three-hour-old replay carrying live moved a stale point to live, and the watchdog put it back on the next tick. That is two transitions and two control-log rows where the document asks for none.
It survived because of the test. test_a_late_replay_does_not_push_the_deadline_out is named for the rule, starts from a live point, and asserts that it goes stale. The sentence is about a point that is already stale, and no test started there. The audit's line on it, at the time:
How it survived: the test that quotes the sentence asserts the half that works.
The fix was wider than the sentence
Holding the quality and applying everything else is what the sentence appears to ask for. The decision row records why that does not work.
A host reconnecting after an outage replays many rows, oldest first. The projection wrote observed_at unconditionally, so an old row moved it backwards, and every deadline is derived from observed_at. Holding only the quality
would let the first row lower the bar the second is judged against, and a few rows in a replayed hour would read as current.
The transition published on the bus also carries the observation's own quality. A held row that was still published would tell every subscriber the point read live while the projection said stale.
So the reading taken was the stronger one: a late replay is written to the log and advances nothing.
It is a row in the history and no part of the present.
The same row names the trap in the obvious implementation. "Older than the current deadline" is true of every healthy observation; a point reporting every 30 seconds under a 45-second window is always inside its deadline. A rule keyed on lateness alone
would freeze the entire projection on the first observation after startup.
Two more rows came out of building it. One out-of-order observation could mark a healthy point stale, for the same reason: observed_at went backwards and took the deadline with it. The other was filed as a cosmetic note about three fields nobody reads. It turned out that a withheld row was still being treated as the predecessor of the row after it, which suppressed a real edge on the next reading.
Then the document moved
The decision gated its rule on the point being stale, because that is the state the sentence is about. Which other states a late replay may promote was left as an open question against the document.
The document's answer was to stop asking what state the point is in. The current text says a reading that was already past its own deadline when it was ingested "is expired on arrival: it is written to the log and advances nothing". It is judged on its own age, measured from ingested_at, instead of against whatever deadline the point happens to hold. That closed the question and superseded part of the decision that had prompted it.
One sentence in the copy published here is now six paragraphs.
Criteria that passed without a core
The first exit criterion in the published copy is specific about how to test it:
Load a checkpointed projection, start the core, and assert zero
restored → staletransitions inside each point's grace, and the correct ones after it.
What existed was six unit tests over a watchdog constructed inside a test. The integration test that does load a checkpoint and start a core asserted nothing about staleness. The second criterion was covered in three pieces, with core.{n}.staleness-suppressed asserted against a stub that returns what it is given. The audit: "Nothing asserts the chain".
Both got the tests the document asked for, against a started core swept by its own timer. The register row for that work:
The second test found a live bug on its first run, which is what the deferral was about
staleness-suppressed is the only boolean among the core's signals about itself. The code that turns a sample into a row narrowed the value to a number and excluded booleans without putting them anywhere. So the signal the published document describes as saying "plainly that this is happening" was written on every pass as a row with no value in any column. The stub had asserted one layer above the row.
The third criterion is the quality propagation table, one fixture per row. It passed. The row that records what was behind it:
Derived points were validated, computed, and neither evaluated nor written. Exit criterion 3 passed on pure functions with no loop around them
Entry one quoted a decision titled A guard with no caller is not a guard, and I wrote that I had no idea how many more of those there were, presumably not zero. This was the next one, and it covered every derived point in the system.
Wiring it found a collision
A derived point's output goes through the ingest writer into the observation table, because rehydration reads that table and a chain of derived points has to be rebuildable from the log.
The dedup key is the one from Ingest Path and Log Schema: instance, epoch, sequence and ingested_at. Every row in one derived pass shared three of the four, and the pass wrote one evaluation at a time with the sequence restarting at zero. Every point got the same key. The insert kept the first and dropped the others silently, and the symptom was a chain whose second point stayed unknown for ever.
No unit test of a single derived point could have seen that. It needed two points written in the same pass.
The stale point whose checkpoint never moved
This one was found in a running core.
Ingest Path and Log Schema gives the checkpoint's write policy as "immediately on quality change". That write was made by the ingest pipeline, from the observation that caused the change.
A point going stale is a point that has stopped sending observations. So the path never ran for it. The checkpoint row sat at the last quality the point had reported, while the core's in-memory count of stale points moved. The two disagreed for as long as the point stayed quiet.
Patch coverage had flagged it. Seventeen changed lines were uncovered on the pull request, and they were the watchdog's entire wiring into the running core. Nothing acted on the flag, because the test job was the only required check. The repository's own note:
A coverage gate that reports and does not block is a gate that tells you afterwards.
The gate now sits inside the test job, and every changed line has to be covered for it to pass. I have written before about a rule being only as strong as what enforces it, and this is the same lesson costing a bug.
Where the document changed
Three things in this step ended with the document amended. A fourth did not.
The fan-out
The Driver Contract promises that a driver dying silently "is caught by its own heartbeat going stale - the core marks the instance unavailable after a grace period". The published watchdog document gives one route to unavailable: an availability event from a driver. A dead driver sends no event, and a missed deadline produces stale. No rule anywhere could perform the promise.
The watchdog document took it, before the step started. A stale instance heartbeat now marks every point that instance owns unavailable. The heartbeat itself stays stale, because "it is the evidence rather than a casualty". And "there is no second grace period": the heartbeat's own interval and grace are the wait. Because the fan-out is triggered by a staleness evaluation, it inherits both suppressions, so "neither a restart nor a slow core can take a fleet unavailable".
The propagation table
The published copy gives six rows as a list. Two of its answers are transitions The Semantic Model forbids, unavailable → stale and unknown → stale, and both are reachable. Row one asks whether an input "has ever been live", and nothing stores history. restored appears in no row, though every input is restored after a restart. And the order decides the answer for an input that is bad having never been live, in a list that does not say it is ordered.
The table is longer now, it is ordered, and it says first match wins. Where a row's answer is an illegal move, the point holds what it has and the control event carries the reason held. That is the two-exit rule on unavailable from entry two, applied one level up.
Flap damping
The published text says transitions "are counted in a rolling window; past a threshold" further ones coalesce "until it settles". It gives no window, no threshold and no definition of settled, and underneath that it does not say whether the window is per point or global. The audit left it unbuilt on purpose:
Building to a guess would put the only absolute duration in the mechanism.
The document now says the window is per point and four times the point's own tolerated silence, the threshold is six, and settled means a whole window with no crossing. That last one is deliberately not "the point is live for a window", because a dead device settles into stale and would never clear.
It has been built since. Building it found that the register row describing the work had named the wrong place to put it. The row said damping was a filter over the watchdog's output, which would have counted entries into stale. A point can legally go stale → bad → stale, so that is a different number from the live/stale crossings the document counts.
The wheel
The published copy asks for a hierarchical timer wheel. The code has a flat one, and the document still says hierarchical. The document names the structure and then says what it is for: one deadline per point, one-second resolution, O(1) reschedule. A flat array of buckets has all three at four thousand points. The difference is recorded as a decision, along with the condition under which hierarchy becomes the answer, and the document has been left as it is.
A seam review, pointed back
Introduction now requires every step to end with a seam review, and step four was the first to end under that rule. The first run of the procedure pointed forward, at documents that had not been built yet. What it found is still reserved for the Mondays after those documents publish.
The second run pointed back at this step, reading the watchdog document against its neighbours. Because the step was already built, its findings arrived with a reading already taken in code. Two of them sit between documents that are both public.
An escape hatch that exists in one document
The Semantic Model, on logarithmic quantities:
Derived points refuse to compute
meanon a non-agg-safe quantity unless the definition explicitly opts intolog_mean
The watchdog document owns the operator set. Its table lists thirteen operators and log_mean is not one of them. Its mean row reads "rejected when the input quantity is not agg_safe", and its exit criteria ask for exactly that refusal. The name appeared in no vocabulary and nowhere in the code.
The code refused unconditionally, which is what the owning document says. The review gave it no decision row, and said why:
That is a choice the code should not have had to make silently.
The fix went the generous way. The operator set gained log_mean, computed as 10 * log10(mean(10^(x/10))), and refused on an agg_safe quantity as the mirror of mean's refusal.
Two columns the vocabulary never had
The published watchdog document says a derived point declares its output quantity, and that a mismatch with "what the operator produces from the input quantity" is a validation failure at load. Checking that needs something that says what a rate over energy is, or a difference of two temperatures.
The amended operator table reads two fields off the quantity vocabulary for it, delta_quantity and rate_quantity. The Semantic Model owns that vocabulary and printed it with four columns. Neither field was one of them.
The registry file declared both, with a rule no document stated: leaving one out means the operator is refused. The decision row has the reason for refusing instead of inheriting:
20 °C minus 21 °C is −1 degree of interval, not −1 °C of anything
The Semantic Model has the two columns now. The amendment got one sentence wrong on the way in, about when an omission is refused, and it was corrected against the code before it was committed. An amendment written to close a finding about drift had drifted from the code itself, by one sentence, before it landed.
A third finding from the same review waits for its document.
What the step has in common
Entry three's findings were all about placement. These are about joins.
The late-replay sentence had two halves and its test held one of them. The exit criteria were met by parts that each worked, with no core running them together, and the dedup key was fine for one derived point and collided as soon as there were two. A stale point's checkpoint depended on the one write path a stale point never takes. The promise about a dead driver sat in one document while the machinery to keep it belonged to another.
Each piece was right when looked at alone, which is why the unit tests passed throughout. What caught these was running the pieces together, and for the first two exit criteria that was the test the document had asked for in the first place.
This Wednesday - The Command Plane. The specification's sixth part: where Sentinel starts writing to devices, with desired and reported held as separate values and a write followed through to a verified outcome.
Next Monday - the Dev Diary. A sixth check has joined the five from last time: an audit that reads every resolved deferral against the code it says was built. Plus the register checks moving out into a tool of their own, including the one that renumbers rows when two branches claim the same id.
Start of the diary: Four Registers and a Drift Check. Why the registers exist at all, and the one rule about where a sentence is allowed to live.
Top comments (0)