Your validator printed the right verdict and exited 0

A CI gate is one line:

aca scan ./attestations

Non-zero exit means "do not merge". That single bit is the whole contract between the tool and the pipeline.

On a directory that contained only forged and expired attestations, that command exited 0.

The tool was not silent about it. It printed, for every file it read, the correct verdict:

✗ agent://planner-v2 -> agent://worker-v3
  ERROR: signature does not match payload - attestation may be forged
✗ agent://worker-v3 -> agent://reporter-v1
  ERROR: expired at 2026-08-30T12:00:00Z (ttl 300s)
✗ agent://reporter-v1 -> agent://auditor-v1
  ERROR: no trusted key for issuer agent://reporter-v1

3 attestations scanned, 3 stale

Every fact a human needed was on stdout. The process boundary dropped it, and the process boundary is the only place a CI system reads.

Where the exit code came from

The multi-file scan looked like this:

@click.option("--fail-on-stale", is_flag=True, help="Exit 1 if any attestation is stale")
def scan(directory, max_ttl, max_skew_seconds, fail_on_stale, json_output, ...):
    ...
    all_valid = True
    for f in attestation_files:
        try:
            result = validator.validate(Attestation.from_dict(json.loads(f.read_text())))
            results.append(result)
            if not result.is_valid:
                all_valid = False
        except Exception as e:
            click.echo(f"ERROR reading {f}: {e}", err=True)
            all_valid = False

    # ... print results ...

    if fail_on_stale and not all_valid:
        sys.exit(1)

is_flag=True means the default is False. So the function's only sys.exit(1) sat behind an opt-in flag; when the flag was absent, the function returned normally and the framework's own default — 0 — became the process status. Nothing in the code said "green"; the exit code was simply the value nobody had bothered to set.

The flag's own name is what makes this a contradiction rather than a design choice: --fail-on-stale reads as "also fail on stale" — as if failing were already the norm and stale were an addition. Stale was the only thing that failed.

The siblings were all fail-closed

The single-file command in the same CLI:

result = validator.validate(attestation)
...
sys.exit(0 if result.is_valid else 1)

The chain checker and the MCP config checker were built the same way. So the multi-file command — the one written to back a CI gate — was the single opt-in exception in a CLI of fail-closed commands.

The half the tests never asserted

The suite already had tests for scan. They asserted the printed text: that ERROR: ... may be forged appears on stdout, that the summary count is right. Every one of them passed both before and after the fix, because the defect lived entirely in the half nobody asserted.

That is the generalisable move for any output-versus-meaning defect: ask where the claim is consumed. A verdict line, a summary count, and an exit code are three claims. Only the third is read by the machine that decides whether the change lands.

The fix: name the exception, not the rule

Fail closed by default; make the unsafe direction explicit and give it a name:

@click.option("--report-only", is_flag=True,
    help=("Always exit 0 and report findings on stdout only. Off by default: a "
          "scan fails closed (exit 1) whenever any attestation is invalid."))
@click.option("--fail-on-stale", is_flag=True,
    help=("Deprecated and redundant: scan now fails closed on any invalid "
          "attestation, not only on stale ones. Accepted for backwards "
          "compatibility."))

…and the two are rejected together, because a flag combination that cannot be honoured is a usage error, not a verdict:

if fail_on_stale and report_only:
    click.echo("ERROR: --fail-on-stale and --report-only are mutually exclusive", err=True)
    sys.exit(2)

...
if not all_valid and not report_only:
    sys.exit(1)
sys.exit(0)

The old flag is kept for one release so the documented CI recipe keeps working — but its meaning moved. Before, "fail the build" was a mode you opted into; now "stay green" is. Defaults should point at the safe direction, and the exception should be the thing that has to be spelled out.

The aftershock: my test proved a different failure

The fix came with ten new tests, 176 passing. One of them built an attestation whose signature was ed25519:ababab... — a signature no key can ever verify. Unfixed, scan exited 0; fixed, it exited 1. That looked like the whole story, and the mutation arm agreed: delete the fix and the new tests go red.

The maintainer merged the PR and added a fifty-nine-line eleventh test to the merged tree. Its docstring is the part worth quoting:

"""The forgery the CHANGELOG entry actually describes.

The test above supplies a signature no key can ever verify, so the
verdict it produces is 'no trusted key for this issuer' - the
*unverifiable* path. That is a real defect class, but it is not the one
this PR claims to fix: it cannot distinguish 'the verifier had nothing
to check against' from 'the verifier checked and the bytes did not match'.
Here the issuer's real public key is trusted and the signature is valid
over the original payload; the attacker then rewrites ``capability`` on
disk. That is the attack the signature exists to stop, it produces the
distinct ``Signature does not match payload`` error, and before the fix
it exited 0.
"""

He was right. Two error strings — no trusted key for this issuer and signature does not match payload — are two different defect classes with two different fixes. The changelog said "forged"; my test exercised "unverifiable". Both exited 0, so both were cured by the same one-line change, and the mutation arm happily confirmed the coverage I claimed. But the test I shipped would keep passing if someone reintroduced the specific forgery path, because it never constructs a valid signature to tamper with.

Assert the case the claim names. If the entry says forged, build a forgery: a real key, a valid signature over the original bytes, and a payload rewritten afterwards. A weaker neighbour is a different branch, and the branch is only the same case if you don't read the error string.

Checklist

• For an output-versus-meaning defect, ask where the claim is consumed: stdout is read by humans, the exit code by the machine.
• A default belongs on the safe direction; the exception gets a name (--report-only), and flag combinations that cannot be honoured are usage errors — exit 2, not a verdict.
• If a flag reads "also fail on X", check whether X was the only thing that failed.
• A returned status is a claim like any other; assert it, not only the text printed beside it.
• Run every new test against the unfixed code — and then check that the test's name describes the failure the claim makes, not its closest neighbour.