What your records cannot tell you six weeks later

A reader of part 1 of my enterprise adoption series described running a small automation in production without a gateway in front of it. What they reported was not a cost problem. Six weeks later they were debugging a silent failure, and could not answer what a given call had cost or who had issued it.

That gap has a shape worth naming. Decision provenance is what I call the part of a record that lets you answer, later, who decided something and under what conditions it was allowed. Cost is the question people accept immediately, because the number arrives on a bill. Attribution is the one that bites later, and by the time it bites, whoever would have remembered has usually moved on.

Part 5 of that series answered the question of what an audit ledger has to contain: who acted, on what, when, who approved it, and which rule allowed it at which version. That part ended by saying that once those columns are in place, answering which rule allowed an action becomes a query rather than a reconstruction. This piece qualifies that claim of mine. The columns are necessary and they are not sufficient, and none of the three ways they fail is fixed by adding more columns.

Not recorded and equal to the default are different facts

Last August I ran the same probe three times against the same model, from the same binary, to measure how stable the readings were. One run passed 67 percent of cases. The other two passed 8.3 percent. Every field in the report header was identical across all three: provider, model, endpoint, governance profile, agent version, run id.

The report was not missing one fact. It could not tell me whether the three runs had measured the same thing, which means it could not tell me whether I had one result and two outliers or two results and one accident. Everything that could have distinguished them sat outside the recorded set, and three of those things were mutable by something other than my code. The sampling parameters were never sent by my code at all, so the effective values came from the model's own configuration file, which a re-pull of the same tag can replace. The model field recorded a tag rather than a digest, so two runs on different weights read back identically. The effective context length was neither set nor recorded. The server version was not recorded.

The scratch directory for the outlier run had already been cleaned up. Those four facts about it are permanently unrecoverable. That is worth stating plainly, because the useful part of an incident like this is not the recovery. There was none. By then the only thing left to decide was how the next run would be recorded.

What I added has three rules in it, and the first one is about absence. When a provenance field cannot be read, the report records null, never an inferred value. An old report with no digest does not mean it ran the weights I have now, and filling that cell with today's digest would turn a gap into a false statement. There is a test that holds this: a report written before the change reads back as not recorded, rather than as an empty block that could be mistaken for a clean run.

Record the effective value, not the requested one

The second rule is that the record has to hold what the system actually did, not what it was asked to do. For context length that means the value the runtime granted rather than the value sent, and when the two differ, the difference itself is the finding.

This is where the exercise taught me something I had not expected. I re-ran the same probe four times under deliberately different machine conditions, including a cold start, and ten header fields came back with zero drift. The header was reproducible. The reason was not the one I had assumed. The effective context length was the same in all four runs because a server-side environment variable pinned it, not because memory negotiation had reproduced. The evidence was that three models with two different declared context lengths, 40960 and 262144, all read back the same effective number, which negotiation would not produce.

So that field records a number and cannot record where the number came from. Move the same configuration to another machine, or let someone change that variable, and the report stays complete, internally consistent, and wrong about what it appears to say. I have hit this once, in one field, so the general form is a hypothesis worth testing rather than a law: a field can be present, correct, and still fail, when the thing that determines its value lives outside the record. What it suggests is cheap to do. For each field in your record, name what determines the value, then check whether that thing is recorded anywhere. The fields that fail this are the ones that will read as legitimate while being wrong.

Something has to notice when the record stops

The second test is not about fields at all. A record that has stopped being written looks exactly like a system with nothing to report.

Part 5 made the neighbouring point, that a field which is declared and read by nothing is a note rather than a control. This is the other half of it, and it is harder to see. There the values are present and unused, so counting fields finds them. Here the writer has stopped, and what is left behind reads as an ordinary quiet period.

My own failure here is not an engineering one. I keep a weekly metrics table for my own publishing, with the columns already defined, and it went unwritten for seven weeks this summer. The table is not decoration. My strategy document carries a stop condition that reads directly off it: four consecutive weeks with no inbound conversations means stop and re-examine the positioning rather than publish more. The duty to raise that alert is written down too, in the prompt for the agent that handles metrics, marked as proactive rather than on request.

Nobody ran it. Because the table stopped, the condition it existed to trigger could not fire, and the weeks that would have tripped it passed without anything being raised. I found the gap on September 22 by auditing the ledger, which is the one path that does not scale.

When I backfilled the table, the entry for those seven weeks is "no record". Not an estimate. An estimate would have been easy to produce and would have made the table look healthy, which is the worse outcome, because a number that reads as data gets used and nobody rechecks where it came from.

The design response, in the part of my work where I did build it, is to make the absence appear in the artifact rather than in someone's memory. When sampling is not configured, the report header does not leave the line out. It prints that the values were inherited from the backend, not repository-controlled, and not reproducible. Forgetting to configure it is visible in the output of every run that follows.

Provenance is identity, not a reading

The third rule is a restriction, and it is the one most likely to be skipped. Provenance never enters the denominator of any rate. Its job is to let two records be judged comparable or not. It does not produce a score of its own. The test for this compares two reports with the provenance block stripped out and requires them to be byte-identical.

The reason to write this down as a rule is that the moment a provenance field moves a number someone is measured on, the record starts being managed rather than written. The restriction exists to keep the record out of anyone's incentives.

The field most decision records are missing

There is one more column that the enterprise version of this argument needs, and it sits one level below the policy version that part 5 asked for.

In the control plane I keep for my own work, the open-source reference implementation of the change-intent governance layer I wrote about earlier this month, a policy decision record requires the mode it was evaluated under, advisory or enforced, and a boolean saying whether the operation was actually blocked, alongside the status. The status itself is one of four values rather than two: pass, fail, unknown, and conflict.

Six weeks after the fact, "this violated the rule" and "this violated the rule, the rule was advisory that week, and nothing stopped it" are different findings, and a record holding only the status cannot separate them. You will read the first and act on it as though it were the second. Unknown and conflict earn their place for the same reason. No rule covers this case, and two rules disagree, are not denials, and an engine that cannot express them will fold them into one for you, in a direction you did not pick and cannot see afterwards. In mine the precedence is explicit in the code: conflict first, then unknown, then fail.

What I have not built, and two tests you can run

Being precise about the boundary matters more here than usual, because this subject invites claims that sound like implementations. The control plane above evaluates configuration and writes decisions. It explicitly excludes provider credentials, secret-manager resolution, and agent execution, so it has nothing to say about what a running session does with a token it already holds. The reader whose comment opened this piece went on to ask what happens when a credential is rotated in the middle of a long-running session. That question lands exactly in the excluded part, and I do not have a shipped answer to it.

Two tests follow from this. Neither is something I have run against a stack other than my own, so they are proposals rather than a validated procedure, and both are cheap enough to try this week on what you already have.

The stop test. Turn off whatever writes your records, in a test environment, and measure how long it takes for something to tell you. If the honest answer is that it surfaces the next time a person goes looking, the record is bookkeeping rather than a control, whatever its columns are.

The split test. Interrupt a multi-step run so that it retries, then read the record afterwards. One task should still read as one unit of work. If the retry opened a second record, attribution has been lost for precisely the call that was worth asking about, and the ledger will look tidier than the history it describes.

If both pass, the reconstruction question is the one to ask next: take a decision from six weeks ago and answer who made it and under which version of which rule, without asking a person who was there. It is worth running while nothing is wrong. During an incident the same exercise only tells you what has already been lost.