DEV Community

Does AI-generated code silently swallow errors? 120 measured generations: every flagged case was a false positive or a documented fallback

Sumitsuke on September 15, 2026

If you gate AI-generated code with linters or a CI rule that hunts for swallowed errors, this experiment suggests the part you actually care about ...

Read full post

howcani howcani • Sep 16

I re-derived your deterministic layer against the frozen corpus before writing anything. One caveat on method: semgrep isn't installed on this machine, so I re-implemented the two Python rule shapes as AST matches rather than running your tool — with the same one-statement anchor (handler body is exactly return <default>) it reproduces exactly your two Python hits, py_parse_int_s3 and py_parse_int_s9. Everything below is computed from raw/ plus results/gt.csv at 18b9a19, so you can disagree with any cell.

The detector's report card is missing its second half. You report the flagged side: 4 candidates, 0 true positives. Your own labels say the corpus contains 4 samples adjudicated problematic_fallback — py_fetch_json_s2/s3/s7/s9 — and the shipped rule set flags none of them, because all four are guard_default, out of scope. So the row is precision 0/4 and recall 0/4. "Zero undisclosed swallows in scope" is exactly right as stated; the number that makes it readable next to the other one is "four adjudicated problems in the corpus, zero detected". As published, "4 flagged, 0 true positives" can be read as nothing was there to find.

The scope hole has a size, and it costs one rule. Same corpus, same tool class: flag any default return that is not inside an except. Over the 60 Python files that fires on 17 — the 4 problems (recall 4/4), the 9 controls, 2 proper, 2 legit_fallback — precision 4/17 = 24%, i.e. one true positive per 4.25 candidates a human has to read. So 100% of the corpus's problems sit in the shape you name as out of scope, and a rule of the same kind surfaces all of them. That is your takeaway #1 with a number attached, and it makes the repair look cheap: the cost of closing the hole is 13 extra candidates to adjudicate, not a different method.

The controls are the densest source of that same shape. 9 of your 20 controls carry a default return by your own labelling (if not numbers: return 0 # Return 0 for an empty list to avoid division by zero), and the guard-scoped rule fires on 9/9 of them while the try/except rule fires on 0/20. So "all 20 controls showed zero try/except and zero candidates" is a scope-bound statement: off-stage is not empty, it holds 9 of your 13 guard_default rows. They are also return 0 rather than return None, which your 7B footnote already names as the (B) problem one more time — documented, and still able to corrupt downstream arithmetic. The arm built to be the negative case carries the shape that can't be verdicted from syntax, which is an argument for your thesis rather than against it.

Your anchor is a feature, and your labels say so — but it is a second boundary, not the same one. The rule requires the handler body to be exactly return <default>. Loosen it to "the handler contains a default return anywhere" and it fires on 5 more samples: py_fetch_json_s6, py_first_line_s2, py_load_json_s4, py_load_json_s8, py_parse_int_s2 — all five have has_log=Y and all five are adjudicated proper. So the strict anchor is, on your own data, a "no log ⇒ silent" test with 5/5 agreement. There are two boundaries around this detector and they are different in kind: the try/except scope is a hole, the one-statement anchor is a correct filter. "Budget for your detector's scope hole" currently covers both, and the numbers say one of them you'd want to keep.

One artefact note. gt.csv's per-sample shape columns are effectively Python-only: subtype and returns_default are filled for 0 of 60 TypeScript rows (the verdicts are filled — 31 proper, 27 not_adjudicated, 2 false_positive). Your reproducibility note discloses the AST-vs-regex asymmetry; this is where it lands in the artefact — a reader computing the same matrix per language, or looking for the same shape in TS, arrives at 27 unlabelled rows.

Happy to hand over the ~30-line detector and the exact cross-tab if you want to fold the recall half into the repo.

Sumitsuke • Sep 16

You went and re-derived it, so the least I can do is check every cell against the same file before answering. I did, at results/gt.csv in the repo, and the parts that live in the labels all come out the way you say:

problematic_fallback = 4, and they are exactly py_fetch_json_s2/s3/s7/s9. All four have subtype=guard_default, so the shipped try/except rule cannot reach them. Recall 0/4 alongside precision 0/4 — you're right, and that row is missing from the article.
9 of the 20 controls carry a default return (returns_default=Y), and 9 of the 13 Python guard_default rows are controls. The negative arm is the densest source of the shape the detector can't verdict. That's a better sentence than anything in my piece.
The strict one-statement anchor is a filter, not a hole. Your five — py_fetch_json_s6, py_first_line_s2, py_load_json_s4, py_load_json_s8, py_parse_int_s2 — are has_log=Y and adjudicated proper, 5/5. I had them collapsed under one "budget for the scope hole" takeaway; they're two different boundaries and only one of them is a defect.
The TS rows are unlabelled for shape: subtype and returns_default are empty for 60/60 TypeScript rows, while human_verdict is filled (31 proper / 27 not_adjudicated / 2 false_positive). Anyone recomputing the cross-tab per language hits that wall, and the artefact should say so.

One number I could not reproduce from the labels alone: your 17. From gt.csv I get to 13 by taking returns_default=Y AND handling=no_try_except (the 4 problems + 9 controls). The remaining 4 — 2 proper, 2 legit_fallback — must come from reading the source rather than the labels, since those rows are marked as handled inside an except. So I can confirm recall 4/4 and the shape of the trade, but not precision 4/17 from the artefact as published. If your detector fires on those four, that's a fact my own labels don't carry, which is itself worth recording.

What I'm changing, and I'd rather say it here than quietly:

The report card gets its second half — 4 adjudicated problems in the corpus, 0 detected printed next to "4 flagged, 0 true positives". As published, mine reads as "nothing was there to find," which is not what the corpus says.
The scope hole and the one-statement anchor get separated. One costs recall, the other buys precision; they were in the same sentence.
The TS labelling gap goes in the artefact note, not just the reproducibility note.

Yes to the ~30-line detector and the cross-tab. If you send it, I'll run it against the same corpus, publish the confusion matrix in full — both halves this time — and credit the recall half to you in the repo and the article.

For what it's worth, this is the second time today the same failure has bitten me from a different direction: a check that only ever sees the cases you hand it, staying quiet, and the quiet being read as an answer. Yours is the sharper instance, because the cases it never sees were sitting in my own labels the whole time.

howcani howcani • Sep 16

You checked the cells, so let me check mine back — starting with one of my own, because it does not survive the same treatment.

My "0/4 and 0/4" mixed two different fours. The article's 4 flagged = 2 TypeScript (ts_load_config_s0/s2, both false_positive) + 2 Python (py_parse_int_s3/s9, both legit_fallback). My recall denominator is the 4 Python problematic_fallback rows. So one side of that sentence is language-mixed and the other is Python-only. Per language the shipped rule is: Python 2 flagged / 0 TP; TypeScript 2 flagged / 0 TP — and TypeScript has zero adjudicated problems, so its recall is 0/0, undefined rather than 0. Written as "precision 0/4, recall 0/4" it looks like one clean 2×2, and it is not one. Precision 0/4 is right; the recall half needs its language attached.

Your question — the 17. You are right that 13 come out of the labels and 4 do not. The 4 are:

sample	handling	subtype	verdict	guard rule fires at
`py_fetch_json_s6`	proper	(blank)	proper	line 13
`py_parse_int_s3`	swallow_cand	(blank)	legit_fallback	line 22
`py_parse_int_s6`	proper	(blank)	proper	line 14
`py_parse_int_s9`	swallow_cand	(blank)	legit_fallback	line 12

Why the labels cannot reach them: handling describes what the handler does, and returns_default says a default return exists somewhere. There are 8 rows that have both a handler and returns_default=Y, and subtype is blank on all 8 — the artefact does not record where that default return sits. My rule fires on 4 of those 8 (the table) and not on the other 4 (py_first_line_s2, py_load_json_s4, py_load_json_s8, py_parse_int_s2), whose only default return is the handler's. So 17 = 13 + 4, and the 4 are exactly "a second, non-handler default return coexisting with a handled try/except" — a shape your column set currently has nowhere to put.

One column closes it. If subtype stopped being blank whenever a handler exists — say guard_default / handler_default / both — then the guard set is subtype in {guard_default, both} and both matrices become computable from gt.csv with no source reading. py_parse_int_s3/s9 would be both, which also records something the current artefact hides: those two rows are the entire overlap between the two rules. Everywhere else they disagree.

The cross-tab, as requested (Python, my re-implementation, not semgrep):

rule	flagged	TP	FP	precision	recall
shipped try/except	2	0	2	0/2	0/4
guard (this)	17	4	13	4/17 = 23.5%	4/4
union	17	4	13	4/17	4/4

Per-sample, guard rule only — sample | handling | subtype | verdict | lines:

py_avg_control_s0|no_try_except|guard_default|not_adjudicated|[3]
py_avg_control_s1|no_try_except|guard_default|not_adjudicated|[3]
py_avg_control_s2|no_try_except|guard_default|not_adjudicated|[3]
py_avg_control_s3|no_try_except|guard_default|not_adjudicated|[3]
py_avg_control_s4|no_try_except|guard_default|not_adjudicated|[7]
py_avg_control_s5|no_try_except|guard_default|not_adjudicated|[3]
py_avg_control_s6|no_try_except|guard_default|not_adjudicated|[3]
py_avg_control_s8|no_try_except|guard_default|not_adjudicated|[4]
py_avg_control_s9|no_try_except|guard_default|not_adjudicated|[12]
py_fetch_json_s2|no_try_except|guard_default|problematic_fallback|[14]
py_fetch_json_s3|no_try_except|guard_default|problematic_fallback|[15]
py_fetch_json_s6|proper|(blank)|proper|[13]
py_fetch_json_s7|no_try_except|guard_default|problematic_fallback|[14]
py_fetch_json_s9|no_try_except|guard_default|problematic_fallback|[15]
py_parse_int_s3|swallow_cand|(blank)|legit_fallback|[22]
py_parse_int_s6|proper|(blank)|proper|[14]
py_parse_int_s9|swallow_cand|(blank)|legit_fallback|[12]

The detector (Python 3.10+, no dependencies) — python guard_default.py raw/python/*.py reproduces the 17 lines and the four line numbers in the table above:

import ast, sys

def is_default(node):
    if isinstance(node, ast.Constant):
        return node.value is None or node.value is False or node.value == 0 or node.value == ""
    if isinstance(node, (ast.List, ast.Dict, ast.Tuple)):
        elts = getattr(node, "elts", None) or getattr(node, "keys", [])
        return len(elts) == 0
    return False

class GuardScan(ast.NodeVisitor):
    def __init__(self, path):
        self.path = path

    def visit_ExceptHandler(self, node):
        return  # handler bodies belong to the try/except rule, not this one

    def visit_Return(self, node):
        if node.value is not None and is_default(node.value):
            print("%s:%d guard-return-default" % (self.path, node.lineno))
        self.generic_visit(node)

for path in sys.argv[1:]:
    GuardScan(path).visit(ast.parse(open(path, encoding="utf-8").read()))

Two scope statements about that table. It is Python only — my TypeScript stand-in for the try/except rule is a regex, not semgrep, and it fires on 5 files where yours fires on 2, so I claim nothing at all on the TS side. And that is not only my limitation: with subtype and returns_default empty for 60/60 TS rows, nobody can compute TS recall from the artefact even if the detector were exact. The rule firing on the 9 controls is the point, not a defect — it is a candidate generator, and the negative arm is where the shape is densest (9 of your 13 Python guard_default rows).

Three things I would keep exactly as you plan them, and one I would add: the column, since it is what lets a reader recompute both halves instead of taking either of us on our word. Credit is yours to give; what I want out of it is that the cross-tab is checkable by someone who has only the corpus.

Your last paragraph is the part I would keep. A check that only sees the cases you hand it, staying quiet, and the quiet read as an answer — that is a nicer statement of the same defect than the one I posted.

Sumitsuke • Sep 16

Ran your detector against raw/python/ before writing this: 17 files, and the four line numbers in your table (fetch_json_s6:13, parse_int_s3:22, parse_int_s6:14, parse_int_s9:12) come out exactly as stated. So the 4/17 I said I could not reproduce from the labels is reproduced from the source, and you were right about why — gt.csv recorded whether a default return exists, not where.

The column is in. build_gt.py now writes guard_default / handler_default / both for every Python row that has a handler (commit 9d7bf7f in the repo, README documents the values). Regenerated gt.csv: guard set = subtype in {guard_default, both} = 17 rows, verdicts 4 problematic / 2 legit / 2 proper / 9 controls — your cross-tab, from the artefact alone. One row to note: py_parse_int_s6 comes out guard_default rather than both, because its handler raises and the only default return is in the else branch; fetch_json_s6 and parse_int_s3/s9 are both, as you said. And yes — s3/s9 being the entire overlap between the two rules is now visible in the file instead of only in your comment.

Your own correction I have taken as well. The article now attaches language: Python 2 flagged / 0 TP / recall 0/4; TypeScript 2 flagged / 0 TP / recall 0/0, undefined. The 2026-09-17 entry in Corrections credits the detector and the column to a reader; if you would rather be named, say so and I will put the handle in.

Still open on my side: the TypeScript rows. 60/60 without shape columns means nobody can do for TS what you just did for Python, and my TS classifier is a regex. That is the next thing to fix in the repo, not something I can close in a reply.

howcani howcani • Sep 17

I ran your fix against the source before answering, and it holds on every point I can check.

At 89bfa35: my detector fires on 17/60 files with the four line numbers I published (13 / 22 / 14 / 12). Your declared set, subtype in {guard_default, both}, is those same 17 rows — the symmetric difference is empty in both directions, and the verdict mix is 4 problematic / 2 legit / 2 proper / 9 controls, identical to the set my detector produces. py_parse_int_s6 as guard_default is right and I would have had it wrong: the handler raises at line 11, so the only default return is the one in the else at 14, and there is no handler default to be "both" with. And the four rows your column calls handler_default are exactly the four my rule does not fire on, which is the part I could previously only assert from reading source.

Then you closed the TS half six minutes after writing that comment (f839873), so let me do the thing you said nobody could do yet — the TS cross-tab. Same disclaimer as last time: semgrep still isn't installed here, so I re-implemented both shapes with tree-sitter.

The guard set reproduces exactly: 11 rows, ts_get_item_s1-s9 + ts_parse_int_s0/s5, symmetric difference empty.
The two TS hits of the shipped rule also reproduce — but from the other rule in the same file. ts-empty-catch hits ts_load_config_s0 L23 and ts_load_config_s2 L24, the comment-only catch in the demo block, which are precisely the two rows a human ruled false_positive. The return-default rule hits 0 files on the TS half.

Which makes the anchor's effect language-dependent, and that is the thing I did not see last round. The anchor says "the catch body is exactly return <default>", and on both sides it excludes exactly the rows that log first:

	strict anchor	loose (any default return in the catch)	what the anchor excluded
Python	2 (`parse_int_s3/s9`, legit_fallback)	7	5 rows, all `has_log=Y` / `proper` (5/5)
TypeScript	0	9	9 rows, all `proper`

In all nine excluded TS rows the catch body is exactly two statements — console.error(...) then return null (or undefined) — so on that half the anchor removes 100% of the return-default catches, by the same rule it uses on Python. So the published "4 flagged (2 Python + 2 TypeScript)" is not one detector's output: the Python two come from the return-default rule, the TypeScript two from the empty-catch rule, and the rule that reaches the corpus's problems is neither. That row's 0 is a union, and each half of it is zero for a different reason.

On the TS work you say is next — the column is in, and the TS cross-tab still reads 0/11 with recall 0/0, and I'd argue that is not a labelling gap. Two numbers from the regenerated file:

The TS half is more adjudicated than the Python half — 37/60 rows carry a verdict against 29/60 — and still contains zero problematic_fallback.
The two halves share only 2 of 6 task families (fetch_json, parse_int). On fetch_json, the single task family carrying all four adjudicated problems on the Python side, TS is 10/10 proper.

So the TS half's positive class is empty by construction of this corpus, and no column, classifier or adjudication round moves recall 0/0. The two honest options I can see: add problem-bearing TS generations, or print the TS half as a control arm rather than a recall arm — it largely is one already (10/10 proper on the shared task, and the only two false_positive rows in the corpus). That is the same shape as the fix you just made, one level up: the Python recall row was missing its denominator; the TS row has neither numerator nor denominator, and the reason lives in the task sample rather than in the schema.

For the record the TS guard set is 9 proper + 2 not_adjudicated (ts_parse_int_s0/s5), so its precision is 0/11 today with two rulings still outstanding — and those two look like the same shape as the other nine, which is what makes the control-arm reading cheap rather than a way of avoiding a verdict.

Please leave it as the reader. Nothing in the cross-tab needs a name to be recomputed, and the entry is more useful if it can be checked without knowing who wrote the sentence. Taking the language split into Corrections is the right home for it — that half was mine and it was wrong.

Sumitsuke • Sep 17

Checked before replying, as before. Both of your new points hold against the repo.

The "4 flagged" row: the two TypeScript hits come from ts-empty-catch (ts_load_config_s0/s2, the comment-only catch in the demo block) and the two Python hits from py-swallow-return-default — scan_and_count.py's hit list says exactly that, and I had been printing the union as if it were one rule. I also read the nine TypeScript catches the strict anchor excludes: every one is console.error(...) followed by return null or undefined, all proper, so on that half the anchor removes 100% of the return-default catches. Both facts are now in Corrections, the README, and the Japanese Lab page.

The control-arm reading I accept, and it is the better description of what the TypeScript half is: 2 of 6 task families shared, and on fetch_json — the only family carrying Python's four problems — TypeScript is 10/10 proper. I have written it that way in Limitations: no positive class by construction of the task sample, so no column or adjudication round can move recall off 0/0. The two outstanding rulings on ts_parse_int_s0/s5 stay open; I agree they look like the other nine, but those are mine to make with the same rubric as the Python rows, not something to settle in a comment thread.

Unnamed it stays — the entry reads "a reader" and the cross-tab checks without knowing who wrote it. Thank you for doing the TS half before I got to it.

Peter • Sep 15

The if-guard scope hole is the more interesting result here than the try/except tables themselves. One cheap, purely syntactic narrowing that doesn't require touching semantics: flag any default-return call site where the caller doesn't branch on it distinctly from a real value of the same type. In your fetch_json example, if every call site does data = fetch_json(url) then uses data with no None check, that's a mechanical signal the None is being silently propagated downstream, regardless of whether the fallback itself is legitimate.

It won't tell you if returning None was the right contract, that's still a spec question. But it tells you whether the contract is actually respected by its callers, which is cheaper than dataflow or call-site analysis and stays fully syntactic. Might shrink the human pile further, especially for the (B)-shaped case: the moment a caller does arithmetic or a lookup on the return value with no None check, that's a detectable structural inconsistency between contract and use, even without knowing whether the contract itself was well-chosen.

Sumitsuke • Sep 16

That call-site test is the part I should have built and didn't. Mine only looked at the definition side — what the function returns on failure — so it can't tell "None is the contract" from "None is leaking." Yours reads the other end, and it needs no semantics: if every call site does data = fetch_json(url) and then indexes into data, the contract isn't respected whether or not it was the right contract.

One thing I should state plainly about what I measured: every flagged case was scoped inside one function. I measured local swallowing — what the function returns on a failure path — not how far the consequence travels through callers. I never crossed the call boundary, so the article says nothing either way about propagation.

I don't have call-site numbers to offer you, and I'd rather say that than estimate. The narrowing you describe is cheap enough that running it beats arguing about it. If I do, I'll publish the count with the script, including the cases where it fires on a legitimate fallback.

Worth flagging that howcani's comment on the same post found the matching gap on the other axis: the corpus contains four samples adjudicated problematic_fallback, all of them guard_default, and the shipped rule reaches none of them — recall 0/4 next to the precision 0/4 I did publish. Your narrowing and that recall row are the same repair seen from two ends, and both of them live in the shape I called out of scope.

Peter • Sep 17

Good instinct to keep the caller-side check semantics-free -- worth being precise about what it can and can't recover on its own, since it maps directly onto howcani's 0/4 recall gap. A pure AST walk only catches the direct, lexical case; it misses the return value getting stored in a variable and dereferenced later, passed to another function, or destructured. The fix that generalizes: don't walk the AST for "is there a None-check between call and use" -- instrument the call site at runtime, log whether the returned value's None-ness was ever checked before its first attribute access, and treat "never checked" as the population-level assertion you'd want on the definition side too. That closes the recall gap on guard_default specifically, because the caller-side runtime check doesn't need to know it's looking at a guard_default at all, only that a nullable return got used without a check.

Sumitsuke • Sep 17

Agreed that the lexical walk stops at the first dereference — anything stored, passed on or destructured is invisible to it, and runtime instrumentation of the call site is the version that does not need to know it is looking at a guard_default. Two things keep me from claiming it here. This corpus has almost no callers: the 60 Python files are isolated functions (one prints), and 49 of the 60 TypeScript files end with a demo block that just console.logs the return — which is "used without a check" by your definition, but it is the generator demoing itself, not a consumer that has to decide what None means. So there is nothing meaningful to instrument and no population to assert over; the check you describe belongs in a real codebase, where it would also give the definition side its denominator. And it moves the question rather than removing it: "never checked before first attribute access" is itself a policy that a documented None-contract can violate legitimately (parse_int returning None on purpose, checked three frames up). I read it as the right tool for the recall half on real code, with the adjudication step still at the end. Not measured here, so not claimed.

Sumitsuke • Sep 18

A correction to my own number above, prompted by another reader who re-counted it: "49 of the 60 TypeScript files end with a demo block that console.logs the return" is the wrong reading. 49 is the number of files with a console.* call outside any catch body, which also sweeps in check()-style helpers and logs inside guards. Files whose last top-level statement is a console.* call — the demo block I actually meant — are 22 of 60 (55 of 60 contain console. anywhere). The point stands with either number: these are the generator talking to itself, not a consumer that has to decide what None means. The number I wrote was still the wrong one.

Peter • Sep 18

Both readings still land on the same conclusion, which is what matters. On the deeper point about instrumentation moving the question rather than removing it — that's real, but it has the same fix as any lint with legitimate exceptions: make 'checked before first use' the default assertion, but let a call site opt out with an explicit, required reason string, the way #noqa or eslint-disable work. That turns 'parse_int returns None on purpose, checked three frames up' from an unmeasurable edge case into a one-line annotation you can grep for during review — and the annotations themselves become the corpus of legitimate deviations, which is more useful than either an unqualified pass or an unqualified flag.

Bert Shim • Sep 15

You had the 20 controls in before the first run. I didn't, the one time it mattered. A check of mine reported an account as blocked, which surprised me, so I ran it against two unrelated accounts and got the same 403 on both. It had been reading whether the session was logged in, not the account. I only thought to add a control because the answer was surprising, and the results that looked normal I never re-ran.

Peter • Sep 16

The 20 pure-computation controls in the article are the same shape as your fix, but one-sided — they show the machinery does nothing on a case that requires nothing, not that it does something correctly on a case that requires action. Your bug is the complement: a check firing is not proof it's testing what you think, only that it produced the expected output. A negative control tells you the detector doesn't cry wolf; you need a positive control with a known-true answer (a deliberately unblocked account, a deliberately logged-out session) run alongside it to tell you it isn't just echoing a fixed signal. Same lesson as the article's own if-guard blind spot: silence isn't validation until you've shown the thing can also speak.

Bert Shim • Sep 16

Hadn't thought to run it the other way. I confirmed it says no when it should -- never checked it also says yes when it should. Same blind spot as the if-guard case, just from the other seat: a check firing stopped me looking, and it shouldn't have.

Sumitsuke • Sep 16

You're both right that my 20 controls are one-sided, and I'd rather record that than defend it: they show the detector stays quiet on inputs that require nothing. They don't show it fires on an input that requires action. A negative control only rules out crying wolf.

I went and wrote up the positive-control version, because I had a live case of exactly bert's failure in my own tools: A JSONL record split in two: U+2028, U+0085, and the separator I missed.

Short version: a guard escapes invisible characters so a JSON line can't be split by the reader. It handled two of them and had been green for four months. Nobody — including me — had asked how big that class is. It isn't "invisible characters": it's the intersection of what the reader splits on and what the encoder leaves raw. For str.splitlines() and json.dumps(ensure_ascii=False) that intersection has three members — the C0 controls get escaped for you, so U+0085 rides through alongside U+2028 and U+2029. The cases the guard covered worked; the scope was two out of three.

The poison test is the positive control: inject a member of the class that must be caught and require red. Before the fix, the U+0085 row came back as 2 lines. That one red is the entire value of the exercise.

Which changes what's missing from my article: not more samples — a poison row per detector. I'll add it as a column rather than quietly fixing it. (Separately, howcani's comment below has the other half of the same bill: the recall row was missing too.)

bert's framing is the one I'm keeping, and I think it's the same failure as mine from the other seat: absence of an alarm treated as evidence of absence. His check fired and he stopped looking; my guard stayed quiet on a character it had never been handed, and I stopped looking. Neither silence nor a green result is evidence until something that should have gone red does.

Bert Shim • Sep 16

Didn't expect that framing to end up in someone else's article. The poison-row idea is the missing half for my case too -- a check that's never seen the failure it's meant to catch isn't proof of anything.

Sumitsuke • Sep 16

It went in because it was the sharper statement of the failure I had just made in my own tool — same defect, other seat. And yes, the poison row is that defect seen from the check's side: a check that has never been handed the failure it exists to catch has only ever measured its own silence.

Glen Allen • Sep 17

The downstream impact of a fallback is an interesting dimension here. Two functions can both return None on failure, but the real risk depends on what happens to that value afterward. If the caller can distinguish “no result” from “the operation failed,” the fallback may be perfectly reasonable; if both states enter the same processing path, the original failure can become almost impossible to recover from. That suggests verification could trace not only the fallback itself but whether failure information remains distinguishable across the call chain. The most dangerous silent errors may be the ones that become semantically invisible several layers after the original decision.

Sumitsuke • Sep 18

That is the dimension this corpus cannot see, and it is worth saying why. The 60 files per language are single functions with no consumer — the only callers are the generator's own demo lines — so "does failure information stay distinguishable across the call chain" has no population here to be measured on. What the data does show is the ancestor of your point: the four problematic rows are the ones where "no result" and "the operation failed" already collapse inside the function (fetch returns None for 404, 500, network error and empty body alike — the gt.csv note on those rows is literally "erases 404/500/network/empty distinction"), so the caller never gets the chance to tell them apart. Your version is the same defect one or more layers later, where it is harder to name and, I'd expect, more common. A tracer for it would need real call chains and a stated contract for what each None means at each layer; the second part is the human step again. Not measured here, so not claimed — but it is the right next question.

Murali Gour • Sep 16

The call-site check is the right next step and it stays syntactic. What it catches is whether the contract is respected by callers. What it misses is whether the contract itself was wrong.

The fetch_json case is the harder specimen: even if every caller checks for None, the function is still conflating "no data" with "fetch failed." Catching that requires checking whether the implementation encodes the distinction the spec requires, not just whether callers handle the return value.

That's where syntactic analysis runs out. Prolog-based reasoning against a stated behavioral contract narrows the human adjudication pile further than call-site checks alone. DataGrout's Invariant is built around this layer for AI-generated code review.

Sumitsuke • Sep 16

Agreed on the split, and it is a cleaner statement of finding 3 than mine: a call-site check answers "do callers respect the contract", and the fetch_json rows fail on a different question — "does the contract distinguish no-data from fetch-failed". Every caller in the corpus checking for None would not change the adjudication of those four.

Where I stop short: for that second question there has to be a stated contract to reason against, and in this corpus there is none — the prompt is the spec, and the four problematic rows are exactly the ones where the model chose a contract the prompt never fixed. Any checker, syntactic or logical, needs that contract written down first; in my data the writing-down step is the human step. I have not evaluated Invariant or any other tool here, so no opinion on it either way.

Murali Gour • Sep 17

The writing-down step being the human step is the thesis stated in its sharpest form. Your corpus makes it precise: the four rows are exactly where the model chose a contract the prompt never fixed, and no post-hoc analysis recovers what was never specified.

What tools like Invariant can do is narrow the pile once the contract exists, which is a different claim than solving the missing-spec problem. Worth being clear about that boundary rather than implying otherwise.

Sumitsuke • Sep 18

Agreed on the boundary, and I would rather state it your way than mine: once a contract is written down, a checker can narrow the pile of callers and shapes to look at; before it is written down there is nothing to check against, and no amount of post-hoc analysis recovers a decision that was never made. In this corpus the four problematic rows are the second case, which is why the fix I can actually show is the boring one — write the default down, then let a syntactic rule enforce that the code matches the sentence. I have not evaluated any tool for the first case here, so I'll leave that claim where it belongs, with whoever measures it.

Suraj Suradkar • Sep 18

This is a really interesting distinction. A pattern can tell you what the code did, but not necessarily why that behavior was acceptable in the first place.

It makes me wonder if the bigger problem with AI-assisted development isn't just detecting bad code, but preserving the reasoning and assumptions behind decisions so they're still available when someone revisits the code later.

Sumitsuke • Sep 18

Agreed, and the corpus has a small, concrete version of your point. The rows that ended up legit_fallback are legitimate for one reason only: a sentence next to the code says what the default means ("returns null if the conversion is not possible"). The four problematic rows have the same guard-and-default shape without that sentence. So the "reasoning" that survived here was one line of prose, and the syntactic rule could only act after a human had read that line against the row. The two TypeScript rows that stayed open until this week are the cautionary half: they had the sentence all along; what was missing was someone reading it. Whether any of this scales to decisions bigger than a default value (why a timeout is 3 s, why a 404 is treated as empty) I have not measured, so I will not claim it.