Does AI-generated code silently swallow errors? 120 measured generations: every flagged case was a false positive or a documented fallback

If you gate AI-generated code with linters or a CI rule that hunts for swallowed errors, this experiment suggests the part you actually care about — "is this return None a contract or a cover-up?" — is exactly the part the rule cannot decide.

I started this project convinced that small LLMs routinely swallow failures — catch an error, return an empty value, pretend nothing happened — and that I'd measure the contamination rate. The hypothesis fell apart in a more interesting way than any clean number would have been. This article is the record of a prior being refuted by measurement, with the collapse documented step by step.

TL;DR

Setup: Qwen2.5-Coder 1.5B (Apache-2.0) via Ollama, CPU-only, Windows 11. 12 frozen tasks × 10 generations (seeds 0–9, temp 0.7) = **120 samples: 100 failure-path functions (50 Python, 50 TypeScript) + 20 pure-computation controls. A 7B robustness footnote adds 30 more (150 generations total, all frozen unfiltered in the repo). Classification: Python by AST, TypeScript by regex; **all 120 labels eyeballed, every swallow candidate and boundary case hand-adjudicated* against docstrings, comments, and the function's contract, published as gt.csv.*

1. A naive Semgrep detector — the kind you might reasonably drop into CI — flagged 4 candidates. Human adjudication: 0 true positives. Correction (2026-09-16): that is only the precision half. The corpus contains 4 samples adjudicated problematic_fallback (py_fetch_json_s2/s3/s7/s9), all of them guard_default, and the shipped rule set reaches none of them — recall 0/4 next to precision 0/4. Correction (2026-09-17): those two fours are different populations. Precision 0/4 mixes languages (2 TypeScript false positives + 2 Python documented fallbacks); recall 0/4 is Python-only. Per language: Python 2 flagged / 0 TP / recall 0/4; TypeScript 2 flagged / 0 TP / recall 0/0 — the TypeScript rows contain no adjudicated problem, so TS recall is undefined, not zero. Read the pair together: "4 flagged, 0 true positives" alone can be read as nothing was there to find, and the corpus says otherwise. Two were false positives (an empty catch in a usage example the model appended outside the function under test), two were legitimate fallbacks, documented (docstring or comment) as the function's contract. But this 0 is not "AI doesn't swallow errors" — my detector targeted try/except and structurally never looked at if-guard default returns, which is where the suspicious shapes actually lived (see finding 3).
2. In this sample, "fails loudly" and "no handling at all" dominated — not swallowing. Of Python's 50 failure-path generations, 39 (78%) used no try/except: 14 raise explicitly via if/else (loud), 4 return None behind an if guard, 21 just let exceptions propagate. TypeScript went the other way: 33/50 (66%) wrote try/catch with log and/or re-throw — proper handling. The only belief that collapsed was my prior that swallowing would be dominant.
3. The core finding: whether code "swallows errors" is not decidable from syntax. The same return None shows up as a documented, role-appropriate contract (parse_int: None if it cannot be converted) and as a hazard (fetch_json: 404, 500 — every non-200 collapses into the same None, while a network error raises instead, a third behavior the caller has to know about — with a comment saying so). A comment doesn't make it sound; a bare guard doesn't make it a bug. The deciding information — the function's role, the caller's expectations, the spec — lives outside the pattern. Static rules can surface candidates; they cannot deliver the verdict.
4. Honest caveats up front: one small model family, 12 tasks, N=10 per task (pseudo-replication — the independent unit is the task, not the generation), single-rater adjudication by me, and a detector whose scope hole I only noticed because the data rubbed my nose in it. Scope: free, local, small models, this corpus. No causal claims about "AI code vs human code" — that comparison belongs to prior work (AIRA, below), not to this experiment.

⚠ Reproducibility scope: generation is nondeterministic and ran once; all 150 generations are frozen unfiltered in the repo. What you reproduce is the deterministic analysis layer — same 120 files in, same distributions and candidate counts out. The final legitimate-vs-swallow labels are human adjudication, published transparently in gt.csv — and the fact that you can't regenerate that layer mechanically is itself the thesis of this article.

Why measure this

AI-generated code quality has a well-cited data point now: AIRA (arXiv:2604.17587, preprint, 2026) ran a deterministic, parser-backed static analyzer over 955 AI-authored and 955 human-authored code samples and reported 1.80× more high-severity findings in AI-authored code (0.435 vs 0.242 per sample), with Broad Exception Suppression (C03) — the "swallowed error" family — as the most frequent check (263 vs 185 in Study 3).

That number gets quoted as "AI code swallows errors." But read the paper closely and it says something more careful, twice:

• Detection is deterministic and works — the paper even warns that an LLM evaluator misses these suppressions at a rate of 44:1 compared to the deterministic scanner. Pattern-matching is the machine's strong suit.
• "Flagged ≠ defect." AIRA's own text states that some fail-soft patterns are contextually intentional and that human review is required before remediation. Two of its checks are permanently human-review-only.

So the open question isn't "can we detect suppression patterns?" (yes, deterministically). It's the step everyone skips: can the verdict — legitimate fallback or bug-hiding swallow — be automated too? I couldn't find a published measurement of that gap (pointers welcome). This article measures it with 120 locally generated functions, a naive detector, and every candidate opened by hand.

Scope declaration up front: this is neither "AI is dangerous" nor "static analysis is useless." It's a measurement of where the machine's jurisdiction ends.

Terms, before the numbers

Three things that look identical in grep output and must not be conflated:

• Default return — on failure, return None / 0 / [] / "" instead of raising. A syntactic shape. Detectable.
• Silent swallow — a default return that erases information the caller needed: failure becomes indistinguishable from "empty but fine." A semantic judgment.
• Legitimate fallback — a default return that is the function's contract ("returns None if unparseable"), ideally documented. Also a semantic judgment.

The whole article is about the gap between the first item and the other two. Classification-wise this sits in CWE-703 (Improper Check or Handling of Exceptional Conditions), with CWE-1069 (empty exception block) and CWE-390 nearby; the exception-antipattern literature (de Pádua & Shang, arXiv:1704.00778, Java/C#) maps the same territory.

Why standard linters don't cover this (and what I built instead)

Checked against current docs, not from memory:

• Python / ruff: S110 (try-except-pass) and S112 exist but are disabled by default — you must opt in via select/extend-select. And a typed swallow like except Exception: return [] sails through anyway.
• TypeScript / ESLint: no-empty only fires on empty blocks, so catch (e) { return null; } is invisible — it's not empty. no-empty even allows a catch containing only a comment. (Note the cultural assumption baked in there: "has a comment" is treated as "is intentional." Whether the comment describes sound error handling is nobody's department. Hold that thought for the counterexample below.)

So I wrote a Semgrep rule that goes after the meaning-shaped pattern — the naive "before" version, verbatim from the repo:

rules:
  - id: py-swallow-return-default
    languages: [python]
    severity: ERROR
    message: "Exception caught and a default value returned without logging/re-raise (silent fallback)."
    patterns:
      - pattern: |
          try:
            ...
          except $E:
            return $R
      - metavariable-pattern:
          metavariable: $R
          patterns:
            - pattern-either:
                - pattern: None
                - pattern: "[]"
                - pattern: "{}"
                - pattern: "0"
                - pattern: "False"
                - pattern: '""'

  - id: py-except-pass
    languages: [python]
    severity: ERROR
    message: "Exception swallowed with pass (silent failure)."
    patterns:
      - pattern: |
          try:
            ...
          except $E:
            pass

(The TypeScript rules are the same shape — catch { return null }, empty catch — and ship in the repo.)

My hypothesis at this point: "run this over N generations, get a contamination rate." Here is where the measurement starts disagreeing with me.

Setup

• Tasks: 12, frozen in tasks/tasks.json before generation. Per language: 5 failure-path I/O tasks (config loading, HTTP fetch, env vars, numeric parsing, file head) + 1 pure-computation control (mean/sum — no try needed). Prompts are neutral: no "handle errors" nudging.
• Task-design disclosure (I did not farm for swallows): pilot runs showed that on plain tasks ("read JSON, return it") the model writes no try/except and nothing interesting happens — which means any "swallow rate" is heavily task-dependent and can be manufactured. I chose failure-inevitable I/O not to inflate the count (the result is ~0 anyway) but to give failure a stage — and kept the pure-computation controls to verify that off-stage, nothing appears. They did: all 20 controls showed zero try/except and zero candidates.
• Generation: Ollama (MIT), CPU-only, qwen2.5-coder:1.5b (Apache-2.0), temperature 0.7, top_k=40 / top_p=0.9, seeds 0–9 per task, threads fixed at 4. The 7B model appears only in the robustness footnote (N=3 per task).
• Determinism probe (before trusting anything): at temperature 0 with a fixed thread count, the same seed generated twice was byte-identical (SHA match). Distributions require temp > 0; hence 0.7.
• Classifier asymmetry, disclosed: Python is classified via ast (a real parse); TypeScript via regex (not a full parser). Cross-language absolute comparisons carry instrument bias — which is why every label was eyeballed and every candidate hand-adjudicated rather than trusting classifier output.

Measured: 2026-07 (corpus generated 2026-07-01). Full per-run metadata in the repo's PROVENANCE file.

Result 1: mostly not swallowing — passing through, or failing loudly

How the 50 failure-path generations per language handled failure:

Reading "no try/except = no handling" would be wrong. AST-splitting Python's 39:

• 14 raise via if/else (e.g. missing env var → raise ValueError(...)) — that's failing loudly, the opposite of swallowing.
• 4 return None behind an if guard — all four in fetch_json (non-200 → return None). A default return! Which my try/except-scoped detector structurally never saw. Remember these four; they're the article's best specimen.
• 21 bare — no failure path written; exceptions propagate to the caller.
• (Check: 14 + 4 + 21 = 39.)

TypeScript's 66% try/catch majority mostly did console.error(...) + throw — textbook handling, no swallowing.

The language difference (Python avoids try/except, TS writes it) is an observation, not a finding: prompt phrasing, language idiom (async/await + try/catch is TS boilerplate), and the AST-vs-regex classifier asymmetry all confound it. The spine of this article is the counterexample below, not this table.

So the original hypothesis — "AI swallows failures at some rate N% I can report" — collapsed in the first table: in this sample, swallowing wasn't the dominant behavior at all.

Result 2: every detector hit was a false positive or documented-legitimate

The naive Semgrep rules flagged 4 of 120 (2 Python, 2 TypeScript). Opening all four by hand:

TypeScript, 2 hits (ts_load_config) = false positives. The function under test was exemplary (code condensed and annotated from the corpus):

async function loadConfig(path: string): Promise<any> {
    try {
        const data = await fs.promises.readFile(path, 'utf8');
        return JSON.parse(data);
    } catch (error) {
        console.error(`Error reading or parsing the file at ${path}:`, error);
        throw error; // logged and re-thrown -- not swallowed (proper)
    }
}
// ...but the model appended a usage example after the function:
(async () => {
    try { const config = await loadConfig('./config.json'); }
    catch (error) { /* Handle any errors... <- comment-only catch */ }
})();

The rule fired on the demo block's catch — (a) outside the function under test, (b) "empty" only because Semgrep's AST ignores comments. Context makes it an obvious false positive; the pattern alone can't know that.

Python, 2 hits (py_parse_int) = documented, legitimate fallbacks. The docstring states the contract — None if it cannot be converted (the second one says the same in a comment).

Bottom line: undisclosed swallowing inside try/except = 0 — and detected problems in the corpus = 0 out of 4 (Python; the TypeScript rows contain no adjudicated problem, so TS recall is 0/0). The first number is precision; the second is recall, and the article originally printed only the first. And now the honest part: my detector's own scope hole. The rules target try/except — the if-guard default returns (the four fetch_jsons) were never in scope. So the corpus does contain default returns; the accurate claim is "zero undisclosed swallows within the detector's scope," not "zero problematic fallbacks in the corpus." Whether those four are problems is precisely the question syntax can't answer — next section.

Result 3 (the core): the same return None, and what separates them isn't in the code

Two functions from the corpus (condensed and annotated). Which one swallows errors?

# (A) numeric parsing: None if unconvertible (contract stated in the docstring)
def parse_int_field(data, key):
    """Returns the int, or None if it cannot be converted."""
    try:
        return int(data[key])
    except ValueError:
        return None            # role-appropriate fallback

# (B) HTTP fetch: None on non-200 (and yes, there's a comment saying so)
def fetch_json(url):
    response = requests.get(url)
    if response.status_code == 200:
        return response.json()
    else:
        return None            # 404 or 500 -- every non-200 flattened into None
                               # (a network error raises instead: a third behavior)

Syntactically, the interesting part is near-identical: on failure, return None. Semantically they're opposites. (A) matches the function's role — "tell me whether this converts" — so None is the answer. (B) permanently destroys the caller's ability to distinguish "no data" from "the fetch failed." And (B) has a comment. Documentation doesn't settle it: a documented return 0 or return None can still silently poison every computation downstream. ("Has a comment = intentional" is exactly the assumption ESLint's no-empty institutionalizes.)

(B) is the strongest specimen this experiment produced: a documented-but-hazardous default return that appeared as a 4-sample cluster (not a one-off), sitting squarely in the blind spot of a try/except-scoped detector — while being exactly the "quietly fails" shape the AIRA numbers gesture at, at population level.

For contrast, non-swallowing code carries its intent inside the syntax:

def get_api_token():
    if 'API_TOKEN' in os.environ:
        return os.getenv('API_TOKEN')
    raise ValueError("The API_TOKEN environment variable is not set.")  # fails loudly

The point, stated carefully: syntactic patterns can surface candidates. The information that separates contract from cover-up — the function's role, the caller's expectations, the spec, and whether the documentation is right — lives outside the pattern. I'm not claiming "undecidable in principle": smarter analysis (types, dataflow, call-site analysis) absolutely narrows the candidates. But "what should this function return, in this context?" is a spec question, and the spec comes from outside the analyzer. Sharper tools shrink the pile; the final reconciliation against intent remains.

Honest scope note: this rests on a small corpus and a handful of specimens — read it as a demonstrated boundary, not a general law about all static analysis. What it demonstrates survives the small N, though, because it's an existence proof: two same-shaped snippets with opposite verdicts, and the verdict-relevant information demonstrably outside the syntax.

What I actually do now: machines generate the candidate list (patterns, distributions, CI notifications — semgrep scan --error exits 1 on candidates, which is fine); a human adjudicates candidates against role, caller, and spec. A green scan is read as "zero candidates for a human to look at," never as "pass."

Where this lands relative to AIRA

Neatly inside its own fine print, it turns out. AIRA detects suppression patterns deterministically and reports the 1.80× population-level difference — candidate generation is the machine's win, and the paper warns that semantic (LLM) evaluation underperforms there, 44:1. But the same paper marks the boundary: flagged patterns aren't necessarily defects, some fail-softs are intentional, human review precedes remediation, and two checks are permanently human-only.

That division of labor is what this experiment probed at specimen level:

• Machines: sensitivity. Surface default-return shapes, chart try/except usage, gate CI with "candidates exist."
• Humans: adjudication. Contract or cover-up — read the role, the caller, the spec.

One misquote I need to preempt, because I nearly published it myself (see pitfall 5): AIRA is not a human-evaluation study, and I am not claiming AI code swallows more than human code — that comparison is AIRA's, made with its own methodology, at population level, in a single-author preprint. This experiment neither confirms nor contradicts it; it maps the adjudication residue the paper explicitly leaves to humans.

Everything I got wrong

In the order I got it wrong:

1. "AI swallows failures; my rule will measure the rate." → Measured: in this sample, swallowing wasn't dominant — pass-through, loud failure, and (in TS) proper handling were. The contamination-rate framing died on the first table.
2. "The detector's 4 hits = 4 swallows." → All four were false positives or documented-legitimate. The honest number is 0, so I report 0 — with its scope. Don't round adjudication up to detection.
3. "The false positives just mean my rule is crude; refinement will fix it." → Refinement did kill the TS false positives. It could not touch the real problem: whether a return None is legitimate isn't decided by any pattern, or even by the presence of documentation. This is where the thesis flipped from "measure the rate" to "the verdict doesn't live in syntax." And auditing my own instrument surfaced the scope hole (if-guard defaults) it had from the start. When your tool's errors persist under refinement, suspect the question, not the regex.
4. The Windows encoding landmine. Semgrep read my TS rule file under Windows' legacy default codepage (cp932) and crashed with UnicodeDecodeError (exit 2) — on an em dash I'd left in a rule message. Fix: ASCII-only rule files + PYTHONUTF8=1 for every run. If your pipeline runs on Windows, pin UTF-8 explicitly or non-ASCII bytes in innocent places will cost you an afternoon.
5. Writing this article, I did the thing the article warns about. My research notes — AI-assisted — said AIRA judged suppressions "by human evaluation." Pre-publication, I went back to the primary source: it's the opposite. AIRA is a deterministic scanner, and it warns against relying on semantic evaluation for this class. I had a plausible summary, I trusted it, and a "verification discipline" article nearly shipped misstating its key citation's methodology. Corrected against the paper itself. Candidates can come from machines — or from your own notes. The final check against the primary source is the human step, and I almost skipped it.

Limitations

• One model family, small models (1.5B main / 7B footnote), 12 tasks, one prompt style. Behavior is strongly model- and task-dependent; no generalization claimed beyond this corpus.
• Statistical independence: N=10 generations per prompt are correlated pseudo-replicates. The independent unit is the task (5 per language), not the generation; don't read the n=50 tables as population estimates. The thesis rides on the counterexample, not the ratios.
• Detector scope: rules and classifiers target try/except; if-guard defaults are out of scope, so "0 swallows" means "0 undisclosed swallows in scope" — the corpus contains default returns whose adjudication is the whole point.
• Promise handlers (added 2026-09-20): promise .catch handlers are outside every pattern in the TypeScript rule file, braces or not — every pattern is anchored on a try { ... } catch clause. In this corpus that is 3 of 48 catch-shaped sites (45 try { ... } catch clauses plus 3 promise .catch handlers): ts_fetch_json_s0 and ts_load_config_s7 block-form, ts_fetch_json_s8 expression-form. All three console.error with no return, so 0 of 3 swallow silently; in each file the function's own catch logs and rethrows while the handler logs and stops, so these three are also the corpus's report-without-propagate sites. The numbers above do not move. promise.catch(() => null) — the form the corpus does not contain — is what the rule would miss. Raised by a reader, 2026-09-18.
• Artefact gap (added 2026-09-16): in results/gt.csv the shape columns subtype and returns_default are filled for the Python rows only — 0 of 60 TypeScript rows carry them (their human_verdict is filled: 31 proper / 27 not_adjudicated / 2 false_positive). Anyone recomputing the cross-tab per language hits 27 unlabelled rows. Raised by a reader who re-derived the analysis from the repo. Partly closed 2026-09-17: for Python rows that have a handler, subtype is no longer blank — it is now guard_default / handler_default / both by where the default return sits (commit 82e5130), so the guard set (subtype in {guard_default, both}, 17 rows) is computable from gt.csv alone. Later the same day (commit f839873) the TypeScript rows got the same three shape columns via tree-sitter, with handling left as the regex verdict the tables above were computed with (the AST agrees on 58/60; the other 2 are the Semgrep false positives). What that exposes rather than closes: the TypeScript guard set is 11 rows (ts_get_item_s1-s9, ts_parse_int_s0/s5) and none of them has been human-adjudicated — their proper comes from the catch logging or throwing, not from anyone reading the default return outside it. TS recall stays 0/0 for that reason, not because the shape is absent — and, more to the point (reader, 2026-09-17), because the TypeScript half has no positive class by construction: the two halves share only 2 of 6 task families, and on fetch_json, the one family that carries all four Python problems, TypeScript is 10/10 proper. Read the TS half as a control arm, not a recall arm. Filling has_raise also moved the four throw-only TS samples from not_adjudicated to loud_fail (same machine rule as Python), so the TS split above now reads 31 proper / 23 not_adjudicated / 4 loud_fail / 2 false_positive.
• Classifier asymmetry: Python AST vs TypeScript regex (nested-brace catches can slip). Human-verified labels backstop it, but discount cross-language comparisons.
• 7B robustness footnote (N=3/task, 30 generations): Python 10/15 no-try, 3 proper, 2 candidates; TS 8/15 no-try, 7 proper, 0 candidates. Both candidates: parse_int again, documented — one returns 0 on failure, which is documented and still capable of silently corrupting downstream arithmetic, i.e., the "(B) problem" one more time. N=3, so no robustness claim — only "not refuted in this range."
• The adjudicator's hole, disclosed next to the detector's: the legitimate-vs-swallow verdicts are single-rater (me), no inter-rater agreement measured. Rubric: (1) does failure yield a default? (2) is that behavior disclosed as intent? (3) does it erase failure information the caller needs? Boundary cases ship with reasons in gt.csv. A second rater is future work — "the last step is human" cuts both ways, so I'm flagging my own last step.
• Conflict of interest: I do AI-code auditing professionally. "The final call needs a human" is a conclusion that favors my line of work. Mitigation is transparency: frozen tasks, unfiltered generations, published adjudications, and the detector's failures disclosed by me, above.

Reproduce it

git clone https://github.com/sumitsuke/ai-silent-defect-scanner && cd ai-silent-defect-scanner
# Deterministic layer: same 120 files -> same distributions & candidate counts
PYTHONUTF8=1 python scripts/classify_split.py      # failure-path distributions (the n=50 tables)
PYTHONUTF8=1 python scripts/scan_and_count.py      # naive Semgrep candidates (the 4 hits)
PYTHONUTF8=1 python scripts/build_gt.py            # regenerate the adjudication layer, results/gt.csv
make scan-mine DIR=/path/to/your/repo              # candidates for YOUR repo (verdicts are on you)

• Mechanical reproduction covers distributions and candidate counts. The final labels are my adjudications, published in results/gt.csv with reasons — by design not machine-rederivable (that's the thesis).
• raw/ ships all 150 generations unfiltered; PROVENANCE records model, quantization, seeds, temperature, thread count, OS, dates. Regenerating gives you a different distribution — that's LLM sampling for you.
• CPU-only, no GPU needed, no paid APIs anywhere in the pipeline.

Takeaways

1. Pattern rules find shapes, and the corpus's actual risk shape (if-guard default returns) wasn't the shape my rule watched. Budget for your detector's scope hole, not just its false-positive rate. Correction (2026-09-16): this takeaway originally covered two different boundaries, and only one of them is a defect. (a) The try/except scope is a hole — it costs recall: 100% of the corpus's adjudicated problems sit outside it. (b) The one-statement anchor (handler body is exactly return <default>) is a filter that buys precision — loosening it adds five samples (py_fetch_json_s6, py_first_line_s2, py_load_json_s4, py_load_json_s8, py_parse_int_s2), and all five are has_log=Y and adjudicated proper, 5/5. Keep (b), close (a).
2. Documentation is not adjudication. A commented return None and a return 0 documented in its docstring both appeared here as disclosed and still hazardous. "Has a comment" is a lint convention, not a verdict.
3. Split the jurisdictions and staff them accordingly: machine = candidates and distributions in CI; human = contract-vs-cover-up against role, caller, and spec. Green means "no candidates today," not "correct."

If you remember one sentence, please don't make it "static analysis is useless" — the detector did its detection job fine. Make it: syntax can surface the suspects, but conviction requires reading intent — and intent isn't stored in the AST.

Detection code, all 150 generations, and the human adjudication layer (gt.csv): github.com/sumitsuke/ai-silent-defect-scanner. Every number above is measured; anything not measured is labeled as not measured.

This is an English adaptation of my Japanese article on Qiita (Qiita is a Japanese dev-blogging platform) — written by me in Japanese, restructured and translated with AI assistance, human-reviewed. If you spot an error, comments and issues are open; I'll verify against the frozen corpus and correct with a changelog.

Verification record (environment, verdict, last verified date, evidence) and the canonical write-up: https://sumitsuke.jp/via/devto/lab/ai-code-silent-fallback/ — code, data and reproduction: https://github.com/sumitsuke/ai-silent-defect-scanner.

Corrections

2026-09-16 — After publication, a reader re-derived the deterministic layer from the repo and pointed out that the report card was missing its recall half. I recounted from results/gt.csv and confirmed every checkable claim: 4 adjudicated problematic_fallback samples, all guard_default, none reachable by the shipped rules (recall 0/4); 9 of the 20 controls carry a default return; the five samples added by loosening the one-statement anchor are 5/5 has_log=Y and proper; the TypeScript rows carry no shape labels. One number I could not reproduce from the labels alone is their precision 4/17 — from gt.csv I reach 13 (returns_default=Y AND handling=no_try_except); the other four would have to come from reading the source. Changes above: the recall row in the TL;DR and in Result 2, the split of scope-hole vs anchor in Takeaway 1, and the artefact gap in Limitations. Nothing in the measured data changed.

2026-09-17 — The same reader came back with two things. First, a correction of their own: "precision 0/4, recall 0/4" reads like one 2×2 but is not — the 4 flagged are 2 TypeScript + 2 Python, the 4 problems are all Python, and TypeScript has no adjudicated problem at all (recall 0/0). I had printed the pair the same way; language is now attached in the TL;DR and Result 2. Second, the 4/17 I said I could not reproduce from the labels: they posted a 30-line AST detector (Return of a default value outside any except handler), and I ran it against raw/python/ — it fires on exactly 17 files, the four "extra" ones are py_fetch_json_s6:13, py_parse_int_s3:22, py_parse_int_s6:14, py_parse_int_s9:12, line numbers as they stated, and the verdicts on the 17 are 4 problematic_fallback / 2 legit_fallback / 2 proper / 9 controls (not_adjudicated) — recall 4/4, precision 4/17. The reason the labels could not reach those four is that gt.csv recorded whether a default return exists but not where; their proposed column closes it, and build_gt.py now writes guard_default / handler_default / both for every Python row that has a handler (of the four: py_fetch_json_s6 and py_parse_int_s3/s9 are both, py_parse_int_s6 is guard_default — its handler raises, and its only default return is in the else branch). Nothing in the measured data changed; one column was added — and, by evening, extended to the TypeScript rows (f839873), which turned the TS artefact gap from "no shape columns" into "11 guard-default samples nobody has adjudicated" (Limitations).

2026-09-17, later — The reader re-ran both fixes against the source (Python 17/60 and the four line numbers reproduce; the TypeScript guard set of 11 reproduces exactly) and added two things I had not stated. First, the "4 flagged" row is not one detector's output: the two Python hits come from py-swallow-return-default, the two TypeScript hits from ts-empty-catch (I confirmed this from scan_and_count.py's hit list); the return-default rule fires on 0 TypeScript files, because its one-statement anchor excludes all 9 TypeScript catches that log first — every one of them is console.error(...) then return null/undefined, and every one is proper. Second, the TypeScript half is best read as a control arm: it shares only two tasks with Python, and the task that carries all four Python defects (fetch_json) is 10/10 proper in TypeScript. (I originally wrote here that the TypeScript half "has no positive class by construction of the task sample" — withdrawn on 2026-09-18, see below; whether it has a positive class is an adjudication outcome, not a property of the sample.) The two outstanding rulings on ts_parse_int_s0/s5 remained open at this point. The reader asked to stay unnamed.

2026-09-18 — The same reader's third round. Two things. (1) The sentence "no positive class by construction of the task sample" was too strong and is withdrawn: the guard filter fires on the same two shapes in both halves; TypeScript's guard rows are 9 proper (machine label: the catch logs or throws) and 2 that were open. (2) I ruled the two open rows, ts_parse_int_s0/s5: no catch in either file, and the docstrings state the default ("it returns null" / "otherwise undefined") — the same rule that made py_parse_int_s3/s9 legit_fallback, so both are legit_fallback (commit in the repo, review_note filled). The TypeScript recall line therefore still has no positive to count, but the honest reading is now "0 of 2 opened, both documented", not "0/0, nothing to open". The Python half is not fully adjudicated either: with subtype ∈ {guard_default, both} and returns_default = Y it is 17 rows — 4 problematic / 2 proper / 2 legit / 9 not_adjudicated. One more correction of my own number in the comments: "49 of 60 TypeScript files end with a demo block" was the wrong reading (49 = files with a console.* call outside any catch); files whose last top-level statement is a console.* call are 22/60.

2026-09-20 — The same reader's fourth round: a scope hole the brace-shaped census could not see. Promise .catch handlers match nothing in the TypeScript rule file, with or without braces; in this corpus that is 3 of 48 catch-shaped sites (45 try/catch clauses plus 3 promise handlers), and all three are console.error with no return (0 of 3 swallow silently; all three log and stop where the function's own catch rethrows), so no number above changes. Stated in Limitations now, next to the if-guard hole, with promise.catch(() => null) as the one-line form the rule would miss.

2026-09-21 — Same reader, fifth round: the 48 had no instrument behind it and reads as 45 to anyone recomputing from the rule's anchor. It is now stated as 45 try/catch clauses plus 3 promise handlers, in Limitations and here; and the three handlers are named as what they are in this corpus — the report-without-propagate sites, where the function's own catch rethrows and the call-site handler logs and stops. No number moves.