Green and worthless: what verifying one small application taught me, when an AI agent wrote most of the code.
The Sanity DEV Challenge wrapped up ...
For further actions, you may consider blocking this person and/or reporting abuse
The useful distinction here is between a test that proves a behavior and a test that merely agrees with its own fixture. For AI-assisted changes I now want an independent observable too: a changed input, a read-back from the real boundary, or a negative case that would fail if the implementation were only cosmetically correct.
One more way a test can lie, when the thing you test is an AI model: the same version, run twice with nothing changed, doesn't always give the same results.
So before I trust that a prompt change helped, I run the old version twice and count how many cases changed on their own. That's my noise floor. If the new version changes fewer cases than that, the "improvement" may just be chance.
It's your negative control, applied to a model: a scale that drifts a pound on its own can't tell you that you lost half a pound.
Respectfully, that isn’t a test lying. Variance between runs is the model doing exactly what a probabilistic system does. The problem is pointing a deterministic test at a probabilistic outcome and then blaming the result. Anyone who’s written tests against timestamps knows how that ends: not if it fails, but when.
Your noise floor is a sensible workaround, but notice what it concedes: a single run-and-compare was never a valid test in the first place. And that’s the real issue. This isn’t one test; it’s several, and they answer different questions.
Is the model responding at all? Deterministic. You don’t care what comes back, only that something does, within your timeout and on your terms.
Are you handling the response correctly? Also deterministic. The model can return literally anything at any time: malformed output, the wrong schema, a refusal, an empty string, a novel. Your code must reach a well-bounded, predictable state every time. Validation, exception handling, fallbacks. That’s where your system’s reliability actually lives.
Is the output any good? That’s evaluation, not testing, and it’s where statistical methods like your noise floor belong, ideally with far more than just two runs.
The test isn’t lying to you. You’re lying to yourself if you think layer three, on its own, is your testing strategy.
Fair point, the variance is just the model, and two runs can miss it.
How many runs you need depends on how small a change you want to see. Running everything that often gets expensive while you iterate on a prompt, though.
A cheaper way: run each version twice, and keep the history. When the two runs of the same version disagree, mark that case as unstable. A change in that case needs more runs before it counts. The extra runs go only where the noise is.
That’s a smart refinement: spend the extra runs where the noise actually is, rather than everywhere. It also fits neatly into a proper evaluation harness, since you’re logging that history anyway.
One point to be aware of though is that two agreeing runs don’t prove a case is stable. A case that flips 30% of the time will still agree with itself in roughly six out of ten pairs, so plenty of noisy cases will be marked stable on the first pass. Keeping the history across iterations helps, because they’ll eventually show their hand. I’d also pin the seed (where your provider supports it) and the model version first, so you’re measuring the variance you can’t remove rather than the variance you can and over time the evaluation becomes more stable.
Cheap where it can be, rigorous where it needs to be. That’s a good balance.
This is a great reminder that having more tests doesn't automatically mean having better testing. I especially liked the point about tests passing for the wrong reasons. Verification needs to challenge assumptions, not just confirm that the code works as expected. That's even more important when AI is writing the code and the tests!
Great piece.
"The agent broke a rule and told me" is such an underrated result: honest reporting turns out to be worth more than perfect obedience.
The paired-assertion rule is the one I'd steal — a test that X has no effect means nothing without the companion proving the effect exists when X is enabled. Your mutation check (delete the clause, watch it finally fail) is the honest version of coverage. Do you run that check across the whole suite now, or only on the tests you already suspect?
Targeted, not suite-wide. I haven't automated mutation testing across the full suite, and the checks I describe were performed manually during the audit.
I started with tests that looked suspicious, but I wouldn't want suspicion to be the permanent selection criterion. Some of the most dangerous vacuous tests look perfectly ordinary until you deliberately remove the behavior they're supposed to protect.
My longer-term preference would be to require positive controls and targeted mutation checks for load-bearing invariants, particularly authorization, state integrity, and irreversible operations. That gives the practice a defined scope without pretending every assertion needs the same level of scrutiny.
That scope is the honest middle — requiring positive controls plus mutation checks only where the invariant is load-bearing, rather than pretending every assertion deserves the same scrutiny. The selection problem you flag is the interesting one though: if suspicion can't be the permanent criterion, what replaces it — is it the invariant's blast radius, or do you rotate the target set so the ordinary-looking tests get their turn too?
I'd use both, but for different purposes. Blast radius would determine which tests get mandatory positive controls and mutation checks. Anything protecting authorization, data integrity, irreversible operations, or a central domain invariant should qualify regardless of whether the test looks suspicious.
But that still leaves the selection problem you identified. We don't necessarily know which ordinary-looking tests are protecting assumptions that turn out to be important. I'd use a rotating sample of the remaining suite, deliberately including tests that have been green for a long time and haven't attracted much attention. When one fails a mutation check, I'd look for other tests built around the same assumption or fixture pattern rather than treating it as an isolated defect.
I haven't implemented that rotation in Cellar, so this is how I'd approach scaling the practice rather than something I've already validated. The original audit taught me that suspicion is useful for finding the first problem, but it's not a reliable coverage strategy.
The goal wouldn't be to mutation-test everything eventually. It would be to make the selection criteria explicit and use what the sampled tests reveal to improve them.
Using both is the honest answer — blast radius for the mandatory set, rotation for everything else. The part I'd underline is treating a failed mutation check as a signal to audit the fixture pattern, not just the one test: that's where a single vacuous test becomes a systemic find. Do you rotate on a schedule, or does a change to a shared fixture trigger its dependents' turn?
I'd favor a change to a shared fixture triggering a review of its dependents, with periodic sampling to catch problems that don't announce themselves through a change. But that's a proposed approach, not something I've implemented in Cellar.
The larger lesson from this audit was that one vacuous test can reveal a shared assumption affecting several others. That's the part I'll carry forward into future projects.
Thanks for the thoughtful discussion!
A fifth one from container scanning: the gate that cannot tell "nothing critical" from "nothing read".
trivy image --exit-code 1 --severity CRITICAL busybox:1.37exits 0 with no warning. Trivy detects no OS in that image, so it scans no packages, and the report has noResultsat all. The signedcosign-vulnattestation carries the same absence, and it reads as clean. A positive control catches it only if the control image is one the scanner can read:alpine:3.10fails on CVE-2021-36159.The clause I added is your "not verified" outcome written into the admission policy: no severity is counted until the report has a
Resultsentry of classos-pkgs. An empty scan is a missing scan. I reported the exit code upstream: trivy discussion #11403That's an excellent fifth example. The distinction between "no critical vulnerabilities found" and "no OS packages examined" is exactly the kind of evidence boundary I was trying to get at.
What's particularly interesting is that signing the attestation doesn't resolve the problem. It can establish the provenance and integrity of the report without establishing that the report actually covered the intended material.
I like your admission-policy requirement for an
os-pkgsresult. It makes successful coverage a prerequisite for interpreting the severity findings, rather than allowing an empty result to masquerade as a clean one.Thanks for sharing the concrete reproduction and upstream discussion. This is a particularly useful example of why verification has to establish what was examined before interpreting what was found.
Your split is the one I would keep: the signature says who produced the report and that nobody changed it since, and nothing about what it read. A Trivy maintainer answered the discussion on 27 October: no warning and no fail-on-empty are planned, since an image with no detected OS is expected for many targets, a
scratchimage with one binary among them. The one trace of the miss is the lineOS is not detected., printed under--debugonly.So the coverage check stays on the consumer's side, written into the policy as a required
os-pkgsresult. The scanner will not say it for you.The proposed-claim fixture is a great example of a test passing for the wrong reason. A useful companion is to mutate the review-state filter and require the test to fail; that checks the test can actually detect the rule it claims to protect.
Exactly, and that's what finally gave me confidence in the repaired test. Removing the review-state filter had to make the test fail. Before the repair, it stayed green because the fixture's recency ordering masked the missing rule.
I ended up finding two more tests with the same underlying weakness, and all three repairs were verified by deliberately removing the logic they claimed to protect. It's a simple experiment, but it changes the question from "does this test pass?" to "can this test actually detect the failure I care about?"
That is exactly the test I trust most: deliberately break the protected rule and watch the test fail for the expected reason. It is a small mutation test, but it catches fixtures that accidentally make a green result meaningless.
The fourth lie - the right test executing against the wrong system because an environment variable silently defaulted to production - is the most insidious failure mode in autonomous workflows, because every assertion passes with total internal consistency.
I ran into this exact boundary in automated test runners. If an agent writes to staging while the application bundle silently falls back to a default production connection, no amount of assertion rigour will catch it. The traditional remedy of inspecting headers or diffing literals still relies on human vigilance.
In my own test harness, the countermeasure that proved robust is decoupling test certification from raw process exit codes and binding runs to an explicit runtime target fingerprint:
By embedding this target fingerprint directly into a signed execution receipt before evaluating assertions, any silent fallback triggers an immediate abort before a single write or read executes.
To address the first lie (the test that cannot fail) and the agentic tendency to silently tamper with oracles to turn a red suite green, two complementary constraints proved essential:
tests/and oracle fixtures during the code synthesis phase. Pairing this with a pre-flight working-tree hash immediately catches whether the agent relaxed an assertion or deleted an edge case to force a pass.Your wine refractometer versus hydrometer analogy hits the core: verification only holds when the observer and the observed fail through completely independent mechanics.
The runtime target fingerprint is a great improvement over my hand-edited literal. Querying the active connection and comparing it against the expected target checks where the application actually connected, not where its configuration claimed it would.
I also like separating the agent's ability to modify implementation code from its ability to modify the tests and oracles that judge that implementation. That's a meaningful independence boundary.
I'd add one qualification to the assertion-count floor: it's useful for detecting missing execution, but the original proposed-claim test demonstrated that an assertion can execute successfully and still prove nothing about its intended rule.
Taken together, these controls address different failure modes: target identity, oracle integrity, execution coverage, and behavioral validity. None replaces the others. Thanks for the detailed example!
An assertion count only confirms that a check executed, not that it had any capacity to fail.
A test can run six assertions and increment the counter, yet prove nothing if the oracle is tautologically bound to the implementation. When an agent (or a fatigued engineer) writes an ineffective test, the failure mode is rarely an omitted assertion; it is circular derivation. The assertion compares an output not against an immutable external fixture, but against an intermediate value derived from the very logic under evaluation. Both sides of the assertion drift together, exit code
0is minted, and the regression passes undetected.In my biobanking systems, where clinical data integrity standards treat unverified test claims as regulatory violations, I borrowed a concept from analytical chemistry to catch this without running expensive full-suite mutation testing: the automated negative control probe.
Every load-bearing assertion is paired with a corrupted input vector during the exact same execution turn:
If the negative probe fails to throw when fed an input with a poisoned checksum or an inverted timestamp, the oracle has zero sensitivity. The runner rejects the test as a vacuous tautology before the build can advance.
Your distinction between target identity, oracle integrity, execution coverage, and behavioural validity is the cleanest taxonomy of this problem space I have read. Each sits at a completely different architectural layer.
That's a useful distinction, especially the difference between an assertion executing and an assertion demonstrating sensitivity to a known defect.
I like the paired negative probe because it makes that sensitivity observable during the normal test run rather than requiring a separate mutation exercise. It also gives the test a much clearer claim: this particular corrupted input must be rejected.
I'd preserve one boundary, though. Demonstrating that a poisoned checksum is detected doesn't necessarily establish that the oracle is independent or that other failure modes would be caught. It establishes something narrower, but importantly, something we can actually verify.
And I think that's where we keep arriving in this discussion: the value of evidence depends on being precise about what it establishes, not simply whether the machinery produced a successful result.
Thanks for bringing the biobanking perspective into this. It's a fascinating application of the same principles.
Your first one has a cousin I keep finding on client sites: an accessibility audit that's green because every image has alt="image". The check asks whether the attribute exists, and nothing in the pipeline ever reads what it says.
Do you have a control for that kind of check, or is that simply the point where a human has to look?
That's a great example of a check answering a much narrower question than the one we actually care about. The pipeline proves that an attribute exists, not that someone using a screen reader gets useful information from it.
I'd approach it with two layers. Automated checks can reject empty or generic values where descriptive alt text is expected, flag repeated descriptions, and identify suspicious patterns. But those are still heuristics. A human needs to assess whether the description communicates the image's purpose in context, including whether the image is decorative and should have empty alt text.
I'd want a known-bad control example, such as a meaningful image with alt="image", that the audit must reject. That at least proves the check can detect the failure it's supposed to catch. It doesn't eliminate the need for human review, but it makes the automated part more defensible.
A known-bad control is such a cheap fix too. One alt="image" left on a staging page on purpose, and the audit has to go red every time, or the audit is the thing that's broken.
The line that hit me: "When one tool writes an implementation and its tests from the same prompt, they share one set of assumptions, and a wrong assumption gets encoded twice and then agrees with itself."
This is terrifyingly accurate, and I just experienced a version of this in technical writing.
I wrote a 1,500-word Python tutorial using a very logical, structured template. I checked it myself, ran the code, and all my "tests" passed. I thought it was perfect. Then an editor at freeCodeCamp read it and told me it read too much like AI and needed to be completely rewritten.
My verification was broken because I was grading my own homework. The editor was the independent oracle.
Your point about "the right test against the wrong system" also hits hard. It's so easy to verify that the code runs, without verifying that the code actually solves the right problem. This is a brilliant breakdown of how dangerous a "green" check can be if it shares the same blind spots as the implementation. Saved!
That's a useful parallel, particularly because running the code and evaluating the tutorial are two different verification tasks. The code can execute correctly while the article still fails to meet the editorial expectations of its audience or publication.
An independent editor brings a perspective that the author, or an AI working from the author's instructions, may not have. It's not quite the same oracle as a software test, but the underlying problem is familiar: we can verify the things we've thought to measure and still miss the requirement that actually matters.
Thanks for sharing the experience, and for reading!
The positive and negative control framing is a much better vocabulary than just calling these weak tests. The first example is the one I recognise most: a test that was measuring the fixture rather than the rule, because the competing claim lost on recency regardless of its review state. Deleting the clause the test claims to cover is such a cheap mutation check, and it is telling that it found two more cases of the same shape. The always-failing grep is the mirror problem I worry about in CI, because once a team learns to dismiss a check as a known false positive, the check has effectively stopped existing. Do you apply the mutation step to every new inertness or alarm test as a routine gate, or only to the load-bearing ones, given the time it would take to do it across a whole suite?
Only the load-bearing ones, at least for now. I haven't made mutation testing a routine gate across the entire suite, and I wouldn't want to suggest otherwise. In this project, I used it where a passing test was supposed to establish an important behavioral boundary, particularly assertions that something cannot affect the result.
The proposed-claim example is a good illustration. The negative assertion was green, but deleting the review-state filter left it green. The paired positive assertion exposed the problem, and mutation confirmed that the repaired test actually depended on the rule it claimed to cover. I found two more cases with the same basic problem.
I think the practical approach is risk-based. If a test protects an authorization boundary, an irreversible operation, or a central domain invariant, I want evidence that it fails when the relevant protection disappears. For less consequential tests, applying that process universally could cost more than it returns.
And I agree about the always-failing grep. Once a team learns to ignore a red check, the check becomes operationally equivalent to one that never runs. Both create the appearance of verification without providing much of it.
I agree
The mutation check is the part I would make non-negotiable, because a test that stays green after its target code is removed is not evidence at all. I would also apply the same control logic to alarms and data checks, since a warning that cannot stay quiet on clean input teaches a team to ignore it. Where do you draw the line in practice between a load-bearing test that deserves this treatment and an ordinary test where the extra control would cost more than it saves?
I draw that line based on the consequence of a false green result, rather than the complexity of the test itself. Authorization boundaries, irreversible operations, state transitions, and rules that protect the integrity of the underlying data are where I'd want mutation checks and positive controls as a matter of course.
For something like presentation formatting or a low-impact helper function, the additional verification may not justify its cost. That doesn't mean those tests shouldn't be reliable, just that I'd prioritize the ones where a misleading pass could cause substantial damage or invalidate a larger set of conclusions.
I also think your point about alarms is important. A warning that fires on everything eventually becomes background noise. Both controls matter: can it detect the condition when present, and can it remain quiet when the condition is absent? Otherwise, we haven't really established what the alarm means.
A React testing version of this: clicking a disabled Save button with userEvent and asserting that onSave wasn't called doesn't test the validity guard inside onSubmit. The click never reaches it, so deleting that guard would still leave the test green.
A valid-input case proves the wiring works. Then test the handler's guard separately: fireEvent.submit(form) reaches onSubmit without native constraint validation getting in the way. That separation makes it clear whether you're testing the disabled UI, browser validation, or the app's own rule.
That's almost a perfect parallel to the proposed-claim fixture. The test passes because something other than the rule under examination prevents the action.
I like the separation you've described: verify that the disabled UI behaves correctly, verify that valid input reaches the handler, and then exercise the handler's own guard independently. Each test establishes a different contract.
It's also a useful reminder that a realistic user interaction and a targeted test of an application invariant aren't always the same thing. Sometimes you need both to establish where the protection actually lives.
The paired-assertion rule deserves to be mechanical: if a test asserts "X causes no change", auto-generate its sibling that enables X and asserts the change exists - if you cannot write the sibling, the first test is vacuous. Mutation testing is the batch version: delete the review-state filter on purpose and count how many of the 530 tests go red. The scary number is not the 3 bugs that slipped through, it is how many of the 530 would stay green with their feature deleted. Did you ever run that count?
I didn't run a suite-wide mutation count, so I don't have that number. The mutations I performed were targeted: remove the review-state filter, confirm the relevant test fails, and repeat for the other cases uncovered during the audit.
I agree with making positive controls a more explicit requirement, although I'd distinguish generating a sibling assertion from proving that the sibling exercises the intended behavior. The original problem was precisely that the fixture made the rule irrelevant, so automatically generating another assertion wouldn't necessarily solve it.
I'd also be careful with the denominator. If removing the review-state filter leaves 500 unrelated tests green, that's not inherently concerning. What matters is how many tests claiming to protect that rule remain green after it's removed.
That's the measurement I'd want from a more systematic mutation-testing pass: not simply how many tests fail, but whether the tests responsible for each invariant can actually detect its absence.
The control framing is what makes this more than a testing cautionary tale. A check with no positive control — nothing proving it can actually detect the thing it guards — produces green results that are really just unexamined results, and a suite of 530 of those feels safer than a suite of fifty honest ones while being the opposite. The detail that three of the four real bugs lived outside the view of the checks designed for their own feature is the sharpest part: the checks were not lazy, they were pointed at a model of the feature, and the bugs lived in the gap between the model and the thing itself.
I think you've captured the uncomfortable part of this really well. The tests weren't necessarily poorly written, and many of them were doing exactly what their assertions said they should do. The problem was that those assertions didn't establish the behavior I thought they established.
That's what made the three missed bugs so revealing. I had a substantial suite and plenty of green results, but the verification model had blind spots that the test count couldn't expose.
The distinction between testing our model of the feature and testing the actual behavior is one I'll be carrying into future projects. Thanks for putting it so clearly.
The date-input example suggests a useful regression-test boundary: exercise the edit sequence, not just the final value. I'd type the year one digit at a time in a real browser, assert that the control doesn't overwrite the in-progress edit, then commit and check the range rule. Setting a complete date programmatically could leave the original failure invisible. I'd also keep picker selection as a separate case, since it reaches the same final value through a different interaction. Did you end up covering both paths after separating editing from validation?
That's exactly the distinction the original failure exposed. Setting a completed date programmatically doesn't reproduce the intermediate states that occur while someone is actually typing.
Separating editing from validation addressed the underlying design problem, but I don't want to claim that I subsequently verified both browser interaction paths unless I've confirmed that in the current test suite.
I agree those should be separate regression cases: incremental keyboard editing, including incomplete values, and selection through the date picker. Reaching the same final value doesn't establish that both interaction paths behave correctly.
Treating positive and negative controls as first-class testing requirements hits the core issue: the wine cellar example showed how easily an assertion can measure the fixture's default state rather than the authority tier resolution logic. When a test verifies byte-identical output without asserting that a lower-tier mutation actually attempted to take precedence, green tests offer zero protection against silent regressions. Do you now use mutation testing or deliberate failure injection to confirm every new check can actually turn red?
That's exactly the distinction I was trying to surface. A passing assertion doesn't establish much if the test never exercised the condition it was supposedly protecting. In the authority-tier example, the important question wasn't whether the output matched the expected bytes, but whether a lower-priority value had actually attempted to override a higher-priority one.
I haven't adopted a blanket requirement that every new check undergo mutation testing or deliberate failure injection. I do think targeted negative controls are especially valuable for critical invariants, though. When a test claims to enforce a boundary, I want evidence that crossing that boundary would make the test fail. That might involve changing a fixture, introducing a conflicting value, or deliberately breaking the implementation and observing the result.
The distinction I'd preserve is between using those techniques to validate the tests themselves and claiming they're part of an automated testing pipeline when they aren't. Both can be useful, but they establish different things.
The fourth lie is the one that scared me most — perfect assertions certifying the wrong cellar because the app silently ignored the dataset variable. 'Verify the target before the result' is the habit I'd steal, starting with just showing the dataset name in the header. Did the hand-edited literal survive contact with a real team workflow, or does someone eventually 'fix' it back into an env var?
That's a fair challenge, and I don't have evidence that the hand-edited literal would survive a larger team's workflow. This was a solo project, so I could make that tradeoff without negotiating with anyone else.
The important thing was eliminating the silent fallback. The environment variable appeared to select staging, the build succeeded, and the application still pointed at production. Making the dataset explicit in the source and visible in the UI made that mismatch harder to overlook.
For a team, I'd want explicit build-time selection that fails on missing or invalid values, plus verification of the dataset actually embedded in the built artifact. I'd keep the visible indicator, too. The invariant isn't that configuration must be hand-edited; it's that the target cannot silently change while every test continues to pass.
Three of four real bugs slipping past every check built for the feature they lived in, that stat stuck with me. When the agent writes both the code and the tests, they can share the same wrong assumption, so green stops meaning anything. Did you ever break a function on purpose just to confirm something would turn red, or did that idea come later?
Yes, but that came later. It wasn't part of my original testing strategy, and that's probably the most important part of the story.
The turning point was a test asserting that a proposed claim couldn't change the cellar's state. It passed, but when I removed the review-state filter entirely, it still passed. The fixture had been constructed so the proposed claim would lose on recency anyway, regardless of whether the review rule existed.
After fixing that, I deliberately removed the relevant logic again to confirm the test would turn red. I applied the same approach to two other tests discovered during the audit.
That changed the question I was asking. Instead of just asking whether the implementation passes the test, I started asking whether the test can actually detect the failure it's supposed to prevent. It's a small wording change, but it makes a substantial difference in what a green result means.
The fourth lie has a cousin on the agent side. In a recorded session of ours, two of twelve domains resolved to the wrong company record and the agent reported all twelve as fine, because every call succeeded. The only thing that caught it was a column nobody asked it to check: the headcount band against the company we knew. Verify the target before the result, same habit.
That's an excellent example of the same failure showing up in an agent workflow. All twelve calls succeeded, but success only established that the tool returned records, not that those records belonged to the companies you intended to retrieve.
I particularly like the headcount-band detail. It gave you an independent signal that the identity mapping was suspicious, even though the agent wasn't asked to examine it. Of course, headcount disagreement doesn't necessarily prove a mismatch, but it gives you a reason to stop and investigate rather than accept twelve successful calls as twelve correct resolutions.
It makes me wonder whether entity resolution should have its own explicit verification step before the agent is allowed to make claims about the retrieved company. Otherwise, every downstream conclusion can be internally consistent while referring to the wrong organization entirely.
This is clearly an instance of a positive-control bug. I executed a version of your first lie in my own testing framework last night. A security policy can only be applied on Windows operating system and therefore one of the tests was written by simply replacing the check with Windows with a test that returns True. This meant that the branch was tested but not the switch used to activate it. Also, the tests executed in the Windows environment do so from within the home folder which is not covered by the security policy. This means that the remaining 173 tests that used the switch did not detect the bug. Flicking the switch brought about no changes. This is clearly a case of positive control failing to work.
That's an excellent example, particularly because there are two independent reasons the tests could remain green. Replacing the Windows check with an unconditional True removed the platform condition, while running the tests from a directory outside the policy's coverage meant they weren't necessarily exercising the protected behavior in the first place.
The detail that stands out is that 173 tests used the switch, yet toggling it made no difference. That's a pretty compelling indication that the tests weren't establishing the behavior they appeared to cover.
I'd be interested in whether moving a representative test into a policy-covered directory and explicitly toggling the switch produces the expected behavioral difference. That would give you both a positive control and evidence that the test environment actually exercises the enforcement boundary.
It's a great illustration of why test count and branch coverage alone don't establish that a security control works.
Absolutely; that is the test that I introduced after this commit (github.com/DeChuks-hub/llmshift/co...). In that test, the actual switch is kept, but the home directory is shifted so that the test directory is the target directory for the rule. A config above that is ignored on Windows and loaded on Linux and macOS. With that configuration, a switch that is always turned off fails on Windows, not passing all 173 tests. Right; it is the number of tests that is deceiving here.
The recency example is the one that stays with me. The review state filter could be deleted and the test still passed, because the proposed claim lost on recency anyway. Nothing was wrong with the assertion. The setup simply made failure impossible.
Pairing every "X has no effect" check with a second test proving the effect exists when X is on is a cheap habit, and the targeted mutation idea from the comments makes it concrete: delete the clause, and if nothing goes red, the test was decoration.
Lie four worries me more on client work. An env var quietly pointing at production turns a green run into a statement about the wrong system. Kevin Hall's headcount band catching two of twelve domains mapped to the wrong company shows the same idea at the data level.
Your line about choosing tests by the consequence of a false green is how I would explain test budgets to a non technical owner. Do you keep a short list of the known bad controls per project?
Not as a formal project-wide catalog yet. In this audit, the known-bad cases were targeted at particular invariants, including removing the review-state filter and confirming that the repaired tests detected its absence.
I do think a small collection of known-bad controls would be useful, especially for authorization boundaries, data integrity rules, and other places where a false green has substantial consequences. It would also make those controls easier to preserve as the implementation evolves.
The important part would be keeping the controls tied to the behavior they demonstrate, rather than collecting fixtures that eventually become another set of assumptions nobody challenges. Thanks for connecting that back to the cost of a false green!
This is an excellent reminder that having a large test suite is not the same as having reliable verification. The distinction between a test that passes and a test that actually proves something is especially important in AI-assisted development, where an implementation and its tests can inherit the same incorrect assumptions and end up validating each other.
The concept of positive and negative controls is particularly valuable. Whenever we test that something should have no effect, we should also demonstrate that the effect can be detected when the relevant condition changes. Similarly, every security check needs a known-clean input to expose false positives and a deliberately unsafe input to prove that the detection mechanism actually works. Otherwise, we risk confusing activity with evidence.
I also appreciate the distinction between testing the implementation and validating the contract. Even a perfectly constructed test can faithfully enforce the wrong requirement. Property-based testing, mutation testing, and independently generated oracles help uncover different classes of defects, but none can completely compensate for a flawed specification. Independent implementations are especially useful because they are less likely to share the same implementation mistakes, although they can still share incorrect assumptions about the expected behavior.
The fourth example highlights another often-overlooked issue: verifying the environment before trusting the result. A test can be logically correct, execute successfully, and still provide false confidence if it targets the wrong dataset, configuration, or deployment. Making the target environment visible and verifying it explicitly should be part of the test contract, not an optional debugging step.
With AI agents, I think the most important improvement is to make uncertainty an acceptable outcome. An agent should be able to report “not verified” when it lacks the evidence needed to confirm a result. That is far more valuable than a confident green check based on an assumption. Verification should rely on observable outcomes, independent evidence, and reproducible experiments—not simply on an agent reporting that its own work succeeded.
Ultimately, tests are not proof that software is correct; they are evidence whose strength depends on the quality of the oracle, the independence of the checks, and the relevance of the environment. AI can accelerate both implementation and testing, but it also makes it easier to produce convincing evidence that is fundamentally misleading.
The question I keep coming back to is this: when an AI agent writes both the implementation and the tests, how often should we require an independent verification path before considering a critical behavior genuinely tested?
Always. 😏
At least for critical behaviors where a false green could have serious consequences. Independence doesn't necessarily require a separate implementation, but it does require a verification path that isn't simply repeating the assumptions of the code being tested.
And that's really the distinction I was trying to explore in the article: a test passing is an observation; deciding what that observation actually proves is a separate exercise.
Thanks for the thoughtful response!
the test that stayed green with the whole review state filter deleted is the failure i see most in generated tests. what changed things for us was a mutation rule in review: delete the clause the test claims to cover, and if it does not go red the test gets rejected. it caught fixtures that were quietly inert more than once. did you automate that mutation pass in CI or is it still a manual check per test?
Still manual, and targeted rather than applied to every test. I haven't built an automated mutation pass into CI for this project.
The first discovery came from deliberately removing the review-state filter and finding that the test stayed green. After repairing it, I repeated the mutation to confirm the test would fail. I then found two more tests with the same underlying problem and verified those repairs the same way.
I like your review rule because it makes the test demonstrate that it actually depends on the behavior it's supposed to protect. For a larger codebase, I'd probably start by applying that requirement to load-bearing invariants and authorization boundaries rather than every assertion, then look at automation where the patterns are repeatable.
The important change for me was treating a passing test as the beginning of the verification question, not the end of it.
The pairing rule is exactly right and deserves to be a lint rule: any assertion of the form 'X has no effect' is vacuous without its sibling 'X, when enabled, does something observable' — I've started writing those pairs as property-based checks (neutrality vs effect) and they catch fixture bugs the same way. The mutation verification is the part to institutionalize: run a cheap mutation pass in CI on changed code paths only, so 'delete the clause and watch it fail' stops being a manual exercise. Injecting 'now' as a parameter is the quiet fix that makes a whole class of time-dependent tests constructible.
I like the idea of applying mutation checks selectively to changed code paths. That seems like a practical way to make the technique repeatable without turning every CI run into an exhaustive mutation exercise.
The property-based pairing is interesting, too, although I'd still want to establish that the generated cases actually exercise the relevant rule. That was the original problem with my fixture: the assertion was reasonable, but the data made the rule irrelevant.
And yes, injecting
nowwas one of those small architectural decisions that made the testing problem considerably easier to reason about.The proposed-claim test is a pattern I'd reuse for any 'agent proposes, human approves' boundary. I'd build the fixture so the proposed claim would win on both tier and recency if it were live. Then the review filter is the only thing keeping it out, and deleting the filter turns the test red by construction. Pair that with accepted (state changes) and rejected (no change, and it can't resurface on the next run). One more risk: when the agent wrote the code and the tests, a vacuous fixture can be a shared blind spot. Have someone who hasn't seen the implementation write or review the fixtures for the load-bearing invariants.
That's a good extension of the fixture design, particularly the requirement that a rejected claim cannot resurface on a subsequent run. It distinguishes the immediate state transition from the persistence of the approval decision.
I also agree with the independent fixture review, especially for load-bearing invariants. When the same agent generates the implementation, the test, and the fixture, the same assumption can be repeated without being challenged.
The key in my original example was making the proposed claim competitive enough that the review-state filter was the only reason it couldn't win. Without that, the test was measuring recency rather than approval.
Giving the agent a third answer, not verified plus the command and expected numbers, fits my eval judges too. Mine still have to pick pass or fail when the evidence they'd need isn't in what they were shown, and I suspect some of my green comes from that rounding up. Once your agent had that option, how often did it use it?
I don't have a reliable frequency to give you, and I wouldn't want to turn a workflow change into a quantitative claim I haven't measured.
The important change was allowing the agent to distinguish between evidence that supports a pass, evidence that supports a failure, and evidence it hasn't actually obtained. Previously, the workflow encouraged a binary conclusion even when the required verification hadn't happened.
The NOT VERIFIED outcome gives it a legitimate way to stop and identify the missing evidence, including the command that should be run and the expected result. But I haven't instrumented that behavior sufficiently to say how often the agent selects it or whether it consistently chooses it when it should.
Your point about eval judges rounding up is exactly why I think that third outcome matters. Forcing a verdict when the evidence is missing doesn't make the evaluation more decisive. It just hides the uncertainty.
the fixture-measuring-itself one keeps biting. agent writes the code and the tests, they share one wrong assumption, green means nothing.
Exactly. When the implementation and tests inherit the same assumption, they can agree perfectly and still be wrong. That's why I found the deliberate mutations so useful: they forced the tests to demonstrate that they could actually detect the behavior disappearing.