Debashish Ghosal

Posted on Oct 15 AI-assisted

No Bootstrap, No Principal: The Day My Audit Log Became Fiction

#ai #testing #controlplane #healthydebate

My v0.1.0 series couldn't talk about authorization, because there wasn't any. v0.2.0 shipped RBAC-lite, scoped API keys, and an encrypted secret vault on HivePlane — and the first thing the field test taught me is that attribution has a cold-start problem.

I shipped a control plane with roles, permissions, and an audit trail — and for an hour, none of it was reachable, because there was no way to mint the first API key.

This is the article about the prerequisite for every governance claim I made in the earlier series: a principal. Without one, your audit log is fiction.

The cold start

The finding was D-10. RBAC was wired. /auth/login existed. Scoped keys were implemented. And the entire model was unreachable end-to-end, because there was no HTTP bootstrap to mint the first admin key. The roles existed; nobody could assume one.

The fix was a dev-grade env admin key — boot the first principal from the environment, then issue scoped keys from there:

# the first principal has to come from outside the API
export HIVEPLANE_AUTH__ENABLED=true
export HIVEPLANE_AUTH__ADMIN_KEY=$(openssl rand -hex 32)
# now you can POST /keys to issue scoped, tenant-bound keys

The lesson is that authn is the admission gate for the operators. Until there's a principal, every "who did this" question is unanswerable, and every other governance feature is theater.

The audit log that named the wrong person

Then I found the worse one. Pause, resume, stop, kill-switch, quarantine — every privileged operator action — was attributed to a client-supplied actor field. The caller put --operator alice in the request. The audit log recorded alice. Any caller could sign an action as anyone.

tip: A who column populated by the request body is a confession that you don't know. The actor must come from the authenticated principal, not the payload. A client-supplied actor is repudiation by construction.

Findings #560, #561, and #541 were all the same shape: the action was real, the attribution was a string the caller chose. The fix is that the principal resolves the actor; the field is read-only. You can attribute an action to yourself. You cannot attribute it to someone else.

The endpoints that had no permission at all

Then the field test found the gaps RBAC was supposed to cover and didn't:

#537 — registry mutation endpoints had no permission checks. A viewer could DELETE /workloads.
#526 — destructive immune-system endpoints (quarantine, kill-switch) were ungated.
#563 — pause/resume/stop had no permission gate; a viewer could stop any run.
#566 — the UI session secret shipped a hardcoded public default.

A role system that doesn't gate the destructive verbs is a role system in name only. The fix is server-side authorization on every privileged action: approve, promote, kill-switch. A viewer cannot approve. A viewer cannot stop. A viewer cannot delete a workload. The 401/403 is the product.

HMAC-first, not bearer-first

The one I didn't see coming was #515. With auth enabled, GitHub and Alertmanager webhooks got 401 before the HMAC signature was verified. The bearer-first model broke event ingest from ≥3 sources — which is a release gate. Triggers are the autonomy layer; if they can't authenticate, the fleet doesn't run itself.

The fix is that webhook ingest is HMAC-first, not bearer-first. The signature is verified before any bearer token is required; the trigger source is trusted by its HMAC, not by a login. Autonomy and authorization have to coexist, or you've secured the plane against its own purpose.

What I learned

Bootstrap the first principal, or none of it exists — the env admin key is the admission gate for operators. Attribution comes from the principal, never the payload: a client-supplied actor is repudiation by construction. Gate the destructive verbs — a viewer who can delete a workload doesn't have a role, they have root. And webhooks are HMAC-first: bearer-first auth breaks event ingest, and autonomy and authorization have to coexist or you've secured the plane against its own purpose.

The v0.2.0 field test asserts per-tenant budget/policy/key isolation with a viewer who cannot approve, and a secret that never appears in logs/traces/agent context (verified by test). Full evidence in the field-test report; the auth bootstrap and key-rotation runbook is at docs/runbooks/auth-bootstrap-and-key-rotation.md.

References

HivePlane — pip install hiveplane==0.2.0
Field test report (36/36)
Auth bootstrap & key rotation runbook
Secrets, RBAC & access audit design
Secrets, RBAC & access audit (User Guide)

Open your audit log. Pick one privileged action. Is the who populated by the authenticated principal — or by a field the caller wrote? If it's the second, your audit log is a story someone else is telling you.

Top comments (10)

howcani howcani • Oct 17

Your fix names the action correctly. It does not decide whether the record of it exists — that is still decided by the same process that performs the effect. I separated the two with a four-arm fixture, with the actor bound to the principal in every arm (nothing reads it from the payload): effect in sqlite, the row in a separate append-only file, the worker killed between the two writes.

effect committed, row after it → dies before the row: action happened, audit rows 0. The log reads clean.
row written before the effect → dies before commit: audit rows 1, nothing happened. The log claims a stop that never occurred.
supervisor records intent before dispatch → worker dies: the sink holds one dangling intent. This is the only arm where the gap has a shape an operator can query.
worker that exits 0 without acting, supervisor stamps completed on exit 0: a completion recorded for a stop that never happened.

So the pair (intent, outcome) is what makes absence visible, and both halves have to be written by a party that outlives the worker — the row cannot be authored by the process whose death is the thing you are trying to observe. That is a different fix from binding the actor, and it survives a correct actor field.

The fourth arm is the one worth running against your own code: exit status is a value the program returns, not a reading of the effect, so a supervisor that stamps on exit 0 records whatever the worker says it did.

The fixture is stdlib only, about 90 lines, and prints the four rows above. Happy to paste it if it is useful for the v0.2.0 write-up.

Debashish Ghosal • Oct 19

This is the sharpest version of the point, and you're right that it's a different fix from binding the actor — it survives a correct actor field. The four-arm fixture is exactly the matrix to run: the "worker exits 0 without acting, supervisor stamps completed" arm is the one I'd run against our own supervisor, because exit status is a value the program returns, not a reading of the effect. Yes please — paste the ~90-line stdlib fixture; it would make the v0.2.0 write-up much stronger.

howcani howcani • Oct 19

Here it is — and I re-ran it before pasting rather than shipping the copy I wrote two days ago, because a fixture that sat in a drawer is not evidence. The output at the bottom is what it printed on this machine just now.

"""Does binding the actor to the principal make the audit row trustworthy?

Four arms, actor bound to the principal in every one (nothing reads it from
the payload).  Stores: `effect` = sqlite the action mutates, `audit` = a
separate append-only file, `sink` = a file the worker never has a handle to.
"""
import json, os, sqlite3, subprocess, sys, tempfile, shutil

# the worker runs in a child process, so the store location must be handed
# over explicitly -- a re-derived tempdir is a different database, and the
# first version of this fixture measured nothing while looking green.
ROOT = os.environ.get("AUDITGAP_ROOT") or tempfile.mkdtemp(prefix="auditgap-")
EFFECT = os.path.join(ROOT, "effect.db")
AUDIT = os.path.join(ROOT, "audit.jsonl")
SINK = os.path.join(ROOT, "sink.jsonl")
PRINCIPAL_ACTOR = "alice"


def init():
    for p in (EFFECT, AUDIT, SINK):
        if os.path.exists(p):
            os.remove(p)
    c = sqlite3.connect(EFFECT)
    c.execute("CREATE TABLE workloads (id TEXT PRIMARY KEY, state TEXT)")
    c.execute("INSERT INTO workloads VALUES ('w1','running')")
    c.commit(); c.close()


def append(path, row):
    with open(path, "a") as fh:
        fh.write(json.dumps(row) + "\n")


def do_action():
    c = sqlite3.connect(EFFECT)
    c.execute("UPDATE workloads SET state='stopped' WHERE id='w1'")
    c.commit(); c.close()


def worker(arm):
    actor = PRINCIPAL_ACTOR            # bound to the principal, never to input
    if arm in ("A", "B"):              # effect first, row after -- crash between
        do_action()
        os._exit(1)
    if arm == "C":                     # row first, effect after -- crash between
        append(AUDIT, {"actor": actor, "verb": "stop", "target": "w1"})
        os._exit(1)
    if arm == "D":                     # exits 0 without doing anything
        sys.exit(0)


def state():
    c = sqlite3.connect(EFFECT)
    s = c.execute("SELECT state FROM workloads WHERE id='w1'").fetchone()[0]
    c.close(); return s


def rows(path):
    return [json.loads(l) for l in open(path)] if os.path.exists(path) else []


def run(arm):
    init()
    if arm == "B":                     # a party outside the worker records intent
        append(SINK, {"actor": PRINCIPAL_ACTOR, "verb": "stop", "target": "w1",
                      "phase": "intent"})
    env = dict(os.environ, AUDITGAP_ROOT=ROOT)
    rc = subprocess.run([sys.executable, __file__, "--worker", arm], env=env).returncode
    if arm in ("B", "D") and rc == 0:  # the supervisor trusts the exit status
        append(SINK, {"actor": PRINCIPAL_ACTOR, "verb": "stop", "target": "w1",
                      "phase": "completed"})
    return {"arm": arm, "worker_rc": rc, "effect_state": state(),
            "audit_rows": len(rows(AUDIT)), "sink_rows": len(rows(SINK)),
            "sink_phases": [r["phase"] for r in rows(SINK)]}


if __name__ == "__main__":
    if "--worker" in sys.argv:
        worker(sys.argv[sys.argv.index("--worker") + 1]); raise SystemExit(0)
    out = [run(a) for a in "ABCD"]
    print(json.dumps(out, indent=1))
    by = {r["arm"]: r for r in out}
    print("\nA: action performed=%s | audit rows=%d  -> log reads CLEAN after an action that happened"
          % (by["A"]["effect_state"] == "stopped", by["A"]["audit_rows"]))
    print("B: action performed=%s | sink phases=%s -> a gap with a shape: attempt with no outcome"
          % (by["B"]["effect_state"] == "stopped", by["B"]["sink_phases"]))
    print("C: action performed=%s | audit rows=%d  -> log claims an action that never happened"
          % (by["C"]["effect_state"] == "stopped", by["C"]["audit_rows"]))
    print("D: action performed=%s | rc=%d | sink phases=%s -> completion for a no-op"
          % (by["D"]["effect_state"] == "stopped", by["D"]["worker_rc"], by["D"]["sink_phases"]))
    if not os.environ.get("AUDITGAP_KEEP"):
        shutil.rmtree(ROOT, ignore_errors=True)

The four arms are the four rows from my earlier comment, and the store layout is what makes them separable: effect.db (sqlite, the thing the action mutates), audit.jsonl (the rows the worker writes), sink.jsonl (the file the worker has no handle to — only the supervisor writes there).

A — effect committed, killed before the row: state stopped, audit rows 0. A stop that happened, invisibly.
B — a party outside the worker records intent before dispatch: state stopped, sink phases ['intent']. The gap now has a shape: an attempt with no outcome.
C — row written, killed before commit: state running, audit rows 1. The log claims a stop that never happened.
D — worker exits 0 without acting, supervisor stamps on exit status: state running, sink phases ['completed']. A completion for a no-op.

For your supervisor the change is one line of authority, not a new component: the completed row must be written from a reading of the effect — query the store, compare the state to what the intent said should change — and never from returncode == 0. Arm D is what happens to the version that stamps on exit status, and it is the arm I would wire first because it costs one query.

Two things this fixture deliberately does not cover, so it doesn't overclaim in your write-up: the appends are plain open(...,'a') with no fsync, so it models a process death but not a machine death; and there is one worker, so no interleaving. Both are the next two arms if you want them.

The AUDITGAP_ROOT line at the top is load-bearing and was learned the hard way: the worker runs in a child process, so a re-derived tempfile.mkdtemp() gives each arm its own database, every row reads zero, and the table still prints green. A fixture whose arms cannot reach the store returns a table, and a table reads like a result.

Arhan Canli • Oct 15

The HMAC-first fix for #515 has two details that decide whether it holds. GitHub signs the raw request body (X-Hub-Signature-256), so the check has to run on the bytes before any JSON parsing, and compare with a constant-time function (hmac.compare_digest, not ==). A valid signature also never expires, so a captured delivery can be replayed unless you store the delivery id (X-GitHub-Delivery) and reject repeats.

The Alertmanager half I'm less sure fits. As far as I know its webhook receiver has no HMAC signing, only http_config (basic auth, bearer, OAuth2, TLS). If that is right, "HMAC-first" for that source means a signing proxy in front of it, or a different check, and a source with no signature falls back to a shared secret that nothing proves came from Alertmanager. Which of those is it in v0.2.0?

On the cold start: ADMIN_KEY from the environment is a root credential that never expires and is visible to anything that can read the process environment or run docker inspect. A one-time bootstrap token that mints the first admin key and then refuses to work again would shrink that window to a single use.

Debashish Ghosal • Oct 19

Confirming the shape: HivePlane signs its own scheme — HMAC-SHA256 over timestamp + "." + nonce + "." + raw_body — verifies with hmac.compare_digest, and blocks replay by claiming (trigger_id, nonce) for a window of max(replay, 2×skew). So it doesn't depend on the provider's signing scheme; a source with no HMAC (Alertmanager) needs the signing proxy or a shared secret, as you say. And yes — the env admin key is break-glass; a one-time bootstrap token that mints the first key and then refuses is the right hardening. Thanks for the precise review.

Kent Bodrov • Oct 15

For queued operator actions, I’d preserve two identities: who authorized the job and which worker executed it. Otherwise deriving the actor from the current principal can replace the human with a service account and lose causality. Does v0.2.0 retain that delegation chain, and what happens if the originating key is revoked before dispatch?

Debashish Ghosal • Oct 19

Good distinction, and it's a real gap: right now the actor is resolved from the authenticated principal (never the payload), and privileged actions are attributed to that principal — but the authorize-vs-execute split for queued work isn't preserved explicitly. If the originating key is revoked before dispatch, the current model would attribute to the executing worker, not the authorizer. Noting that for the write-up; it's the right next step.

Kent Bodrov • Oct 19

I’d test revocation after enqueue and before dispatch. The record should still show who originally authorized the job, but dispatch should fail if that authority is no longer valid. The worker’s own identity must not silently turn a revoked instruction into a new authorization. Historical attribution and current permission answer different questions.

Dhruv Malaviya • Oct 15

"A who column populated by the request body is a confession that you don't know" belongs in every API review checklist. Repudiation by construction is exactly right , and it passes every test, because the tests supply the actor too.

Two additions. The env bootstrap key is the correct fix and needs a rotation story attached, otherwise the break-glass credential becomes the permanent one. That's how a cold-start solution turns into next year's finding.

And #537 is the more common shape. Per-endpoint permission checks are easy to add and easy to forget; deny-by-default at the router with explicit grants fails closed when someone adds an endpoint and doesn't think about it.

Debashish Ghosal • Oct 19

Thanks — and agreed on rotation: the env key is break-glass, and the runbook rotates to scoped, tenant-bound keys immediately. On #537 the fix is server-side authorization on the privileged verbs (registry mutation, approve, promote, kill-switch). You're right that deny-by-default at the router with explicit grants is the stronger shape — it fails closed when someone adds an endpoint and doesn't think about it.