The most expensive thing an autonomous coding agent can produce isn't a broken build. A broken build is free: CI goes red, you move on.
The expensive thing is a green diff that is wrong. It costs the review cycle, the merge, the deploy, and then it costs the afternoon three weeks later when someone bisects it.
And the way an agent produces that artefact is depressingly simple. It writes a change. It runs the tests. One test is inconvenient. It adjusts the test. Everything is green. It says done.
Nothing in that sequence is a lie, exactly. Every step is something a tired human does too. The difference is that when a human does it, another human reads the diff and asks why did this assertion change? When an agent does it at three in the morning across a task queue, nobody asks.
So the question I got interested in isn't "can AI write code?" That's settled. The question is:
What would an AI-generated change have to produce before a human is entitled to trust it?
I found a project trying to answer that concretely, contributed three PRs to it, and got taken apart in review in a way I'm still thinking about. This is what I learned.
Code generation is not verification
Here's the shape of real engineering work:
The middle is not a ceremony. Every stage it skips is one where a specific class of defect is caught.
Consider a ticket: "Fix the crash when a task title contains non-Latin characters." (This is a real class of bug — v0.2.4 of the project I'll describe fixed exactly this on Windows.)
An agent produces a diff. The diff touches an encoding path. Tests pass. Ship?
You can't answer that, because you don't know:
- Did a test exist that reproduced the crash? If not, what exactly demonstrates the fix?
- Does that test fail on the old code? If it passes on both trees, it proves nothing about the bug.
- Did the test count go down? A change that fixes a bug and quietly deletes four unrelated tests is two changes.
-
Did any assertion become tautological?
assert result == resultis green forever. - Did an unrelated behaviour change? Encoding fixes love to move line endings.
Notice that none of those is questions about the code. They're questions about the evidence surrounding the code. And an agent that scores its own work has a structural incentive problem: the thing producing the change is also the thing attesting to it.
A plausible diff is not proof. It's a hypothesis with good syntax highlighting.
Enter no_human
no_human is an open-source (MIT), locally-run workflow that takes a ticket through implementation and review and stops at an open pull request. It runs on your machine, on your own Claude credential, against your own checkout. There is no hosted service in the middle of the loop.
Its README describes what it is in one line: from ticket to reviewed pull request. The more interesting framing is what it refuses to do — it never merges. The orchestrator opens a PR and parks at awaiting_approval. Merge is a human command, always.
The project's own documented pipeline:
ticket ──► context ──► plan ──► implement ──► review ──► test ──► PR ──► you merge
│ │ │ │
│ │ │ └── local runner + optional CI
│ │ └── fresh-context reviewer, edit tools refused
│ └── Agent SDK, your credentials, your checkout
└── grep, git log, past sessions
Two things about that diagram are load-bearing and easy to skim past.
Git is not the model's. Branching, committing and pushing are done by no_human's own git code, not by the agent. Implementation runs behind a PreToolUse hook that enforces forbidden paths, protected branches, a merge ban and a destructive-shell circuit breaker. The agent writes files; it does not drive version control.
The reviewer is not the author. It is a separate Agent SDK session with fresh context, a different model tier by default, and the file-edit tools refused.
The gates, and what each one is actually for
This is the part worth reading even if you never install the thing, because each gate exists to defeat a specific failure mode of autonomous work.
Tamper guard — deterministic, no model involved
testing/tamper_guard.py diffs test files separately from product code, and fails on:
- a net drop in test count or assertion count
- a net increase in
skip/xfailmarkers - a real assertion replaced by a tautology
- a behaviour-faking
autousefixture appearing in aconftest.py
No model judgement. It runs before the expensive review, so a self-gutted suite gets blocked before a reviewer token is spent. It covers Python, JS/TS, Java and the e2e/ tree.
Why this matters generally: if your verification mechanism is inside the agent's write surface, it is not a verification mechanism. Any autonomous system that can edit its own tests needs something outside the loop watching the tests themselves.
Reproduction gate — proving the bug existed
testing/repro_gate.py takes the tests the coder offers as evidence, copies them into a worktree at the merge base, and requires them to fail there and pass on the new tree.
reproduction test
OLD TREE (merge base) → FAIL
NEW TREE (the change) → PASS
A bugfix whose test also passes on the unfixed code has demonstrated nothing about the bug. This is the single cheapest idea in the project and the one I'd steal for a human team tomorrow.
Accuracy note: the default mode is advisory, which still enforces for a Python bugfix where the coder's edits reached .py files through the edit tools. repro_gate.mode: required drops the conditions and enforces for every change. That nuance is in the docs, and it matters — a .py file written by sed isn't bound by the default.
Independent adversarial review
review/reviewer.py opens a fresh session — it has not seen the coder's reasoning, its rationalisations, or its plan. Its Write, Edit, NotebookEdit and MultiEdit tools are refused, along with direct git and forge write commands. By default it runs on a different, higher model tier than the implementer. It is told to refute "done".
What it returns is the part I care about: a checklist of findings with file, line and severity. A boolean verdict. Never a numeric self-score.
Three properties make that verdict hard to game, and all three are code rather than prompt text:
- Every cited location is checked against the actual tree; a finding citing a line that doesn't exist is demoted to advisory.
- The pass/fail is recomputed deterministically from the checklist, not taken on the model's word.
- A reviewer that crashes, times out, or emits nothing parseable fails closed.
That last one is the tell. A system that fails open on a broken reviewer doesn't have a reviewer; it has a formality.
Held-out tests, verifiers, and merge policy
-
tests/held_out/are run by the orchestrator and handed to the reviewer as evidence the implementer never saw. -
.no_human/verifiers.yamllets you write project rules as plain-English statements scoped by path glob; each is put to an independent bounded judge call, and every verdict is recorded — pass or fail, never only the failures. A judge that reaches no verdict after one retry is recorded asunavailableand treated as advisory, on the explicit reasoning that a gap in the gate is not evidence about the change. -
core/merge_policy.pyevaluates a fixed rule set (review passed, tests passed, tamper clear, repro passed or not required, verifiers satisfied, CI success or unknown) into a singleready: boolplus a per-rule breakdown. If the diff edits the merge policy file itself, that's flagged and forcesready: false— a coder cannot author its own merge gate.
Honest stopping
The loop is bounded: 3 attempts per loop, 500 turns per attempt, 9 lifetime attempts across resumes. Repeating an identical tool call, or the same error signature twice, trips stuck detection, which resets context instead of stacking corrections on a confused session.
When it runs out, it doesn't invent a plausible diff. It classifies the blocker into one of eleven categories (MISSING_ACCESS, AMBIGUITY, SCOPE_EXPLOSION, IMPOSSIBLE, QUOTA, BUDGET_EXHAUSTED, and five more) and either parks with a wake condition or escalates with a structured report and one specific question.
The docs put it better than I can:
An honest escalation costs a minute to triage. A confident wrong diff costs an hour to review.
The two workflows, side by side
The common shape:
Ticket
↓
Agent generates code
↓
Agent runs tests ←──── agent can edit these
↓
Agent says "Done" ←──── agent grades itself
↓
You review a diff with no evidence attached
Failure modes: deleted tests, new skips, tautologies, a fix that never reproduced the bug, a confident answer where "I don't know" was correct.
The no_human shape:
Ticket
↓
Context (grep, git log, past sessions)
↓
Plan
↓
Implement (git owned by orchestrator, not the model)
↓
Tests + held-out tests
↓
Tamper guard ← deterministic, before any reviewer spend
↓
Reproduction gate ← fails at base, passes on head
↓
Deterministic evidence (lint, wiring, net-new types)
↓
Independent reviewer, fresh context, edit tools refused
↓
Merge policy → ready: bool + per-rule breakdown
↓
PR opens, task parks at awaiting_approval
↓
YOU approve
The agent's opinion of its own work appears nowhere in that chain as a gate.
The part I keep thinking about: confidence vs. evidence
Every agent I've used will tell you it's confident. Some will give you a number.
That number is a model output. It was generated by the same process that generated the code, conditioned on the same context, carrying the same blind spots. Asking it how sure it is is asking the defendant for the verdict.
Compare:
Reproduction test
OLD TREE → FAIL
NEW TREE → PASS
Test integrity
tests: +58 assertions: +0 net loss
skips: +0 tautologies: 0
→ CLEAN
Reviewer (fresh context, different model, edit tools refused)
findings: 0 blocking, 2 advisory (file:line cited, verified against tree)
→ PASS
Merge policy
review ✅ tests ✅ tamper ✅ repro ✅ verifiers ✅ ci ✅
→ ready: true
Every line there is something a second engineer can re-run. None of it requires believing anything the agent said about itself.
Confidence is a claim. Evidence is a claim someone else can check.
That's the whole idea, and it's not really about AI.
Contributing to a project that is itself about trustworthy AI
Here's the recursive part that made this fun: when the project is a verification system, your PR gets verified by the philosophy it implements.
I contributed three changes, all merged, all under issue #114 — "Attach net-new type-checker diagnostics as review evidence; measure before adopting LSP navigation." The premise of that issue is that the review gate runs on machine-checkable signals, and type diagnostics were a missing one.
PR #164 — net-new type diagnostics as review evidence (merged)
The problem. A reviewer looking at a diff has no way to know whether the change introduced type errors. Running a type checker on the head tree is useless: a repo with 400 pre-existing errors reports 400, and the signal drowns.
What I built. If the repo under review configures a checker (pyrightconfig.json / [tool.pyright], [tool.mypy] / mypy.ini / setup.cfg [mypy], or tsconfig.json), the same checker runs with the same argv over the whole project twice — once at the merge base, once at the reviewed commit — and the base result is subtracted. A repo with 400 pre-existing errors reports net-new: 0.
Four design decisions, each avoiding a specific defect rather than adding a feature:
- Not scoped to changed lines. Unlike lint evidence. The characteristic net-new type error appears at a call site the diff never touched — narrow a parameter and every caller lights up. A changed-line filter discards exactly the diagnostics worth having.
-
Fingerprints ignore line numbers. One inserted import shifts everything below it. The key is
(path, code, digit-normalised message)as a multiset. -
Silence means "did not run", never "clean." A missing binary, a crashed checker, unparseable output — all yield
ran=False, which renders no section. A run that happened and found nothing rendersnet-new: 0, which is a different and usable fact. -
Coverage is reported, not assumed. A throwaway worktree carries no installed dependencies, so
tscandpyrightdegrade unresolved symbols toAny.unresolved_importsrenders as aCOVERAGE LIMITline, sonet-new: 0can never read as an unqualified clean.
Review feedback — three blockers, all real. This is where the project taught me something.
The egress gate went red. Every exec or network channel in the repo is declared in
tests/test_egress_allowlist.py. My checker built its argv fromshutil.which, so the scanner couldn't name the program. The maintainer's point wasn't "add an entry" — it was that the PyPIpyrightpackage is a launcher that downloads Node on first run, so a review could make a network call, and the entry had to say so. Writing it exposed a contradiction in my own module docstring, which claimed "nothing is ever installed."My checker wrote into the tree under review.
mypydrops.mypy_cache/where it runs;tscwrites*.tsbuildinfo. The collector sits inside the window the orchestrator brackets withreviewer_worktree.snapshot/.compare— so a cache write makes that compare report an added path, the orchestrator chargesreviewer_wrote, reverts, and replaces a real verdict with an integrity failure nobody caused. I had reasoned about the collector in isolation instead of about where it sits in the orchestrator. That's exactly where it bit.
The fix was symmetric rather than per-checker: _run_at_commit now serves both sides, so nothing the checker writes can reach the attempt's tree, and untracked files can no longer read as net-new. Pinned by a test using a fake checker that really does create .mypy_cache/missing_stubs where it runs. Reverting the after-run to the reviewed tree turns six tests red.
-
Cost. One deadline for the whole collection rather than a per-run cap (whose real worst case was checkers × sides × cap), off the event loop via
asyncio.to_thread, skipped entirely on the single-turn route.
The thing I'm proudest of is the one nobody asked for. Making the runs symmetric didn't fix the TypeScript coverage problem — it changed its shape. Both sides now degrade equally, so the comparability check passes and the subtraction runs over two blind analyses. Arithmetically sound. But net-new: 0 would have been a false clean, which is the one thing that module exists not to emit. So the COVERAGE LIMIT line stays, and dropping it turns a test red.
And I reported the unflattering number. Detection across 130 local repos: it would render on 32 (25%), of which 94% are the degraded TS path, with a third to two-thirds of what it reports there being unresolved-import noise. I also stated the two limits on that sample — one developer's machine, skewed to TS frontend work, so 25% is not a population figure. It landed as advisory evidence precisely because the coverage line says so out loud.
PR #221 — per-edit type feedback to the coder (merged)
The idea. Phase 1 tells the reviewer. Phase 2 tells the coder, in the same turn that caused the diagnostic. A PostToolUse hook beside the existing lint hook: after an Edit/Write to a .py/.pyi file, run the configured checker on that one file and report what it did not report on that file at its previous successful check. Off by default.
The blocker that mattered most was a sentence, not code. My header said "Your edit to X introduced N diagnostics." The maintainer reproduced two ways it's false:
-
mypy <file>follows imports. He drove the hook with anapp.pythat was md5-identical before and after whilelib.pygained an error — and got told his edit toapp.pyintroduced it. -
Bashisn't in_EDIT_TOOLS, sosed -i,patch,ruff --fixandblackare invisible, and whatever they broke surfaces on the nextEdit, attributed to that edit.
Neither is a false clean, so neither is the unacceptable outcome. But it's a false signal carrying an imperative, delivered mid-turn, on sequences a real attempt produces constantly. A coder that spends turns fixing something it didn't cause is the exact attempt cost the project is trying to avoid.
The block now reads "N diagnostics on or reachable from X that were not reported when this file was last checked" — naming both ways it can mislead, in the text the coder actually sees.
What that review round taught me: the lint hook can honestly say "your edit to X introduced" because ruff sees one file. A sentence that's true for one component is not automatically true for its neighbour. I had inherited the wording along with the architecture.
Two other things went in that I hadn't been asked for:
-
A credential scrub. The checker subprocess inherits the process environment, and
mypyimports whatever a repo'splugins =line names — so a reviewed repo could run its own code with our OAuth token in the environment._checker_envdrops every secret-shaped variable and keepsPATH,HOMEand proxies. It reaches the already-landed Phase 1 too. And because that scrub costs something real (a plugin that readsDATABASE_URLno longer loads), the failure renders aTYPE EVIDENCE: NOT COLLECTEDline naming the cause and ending: "This says nothing about whether the diff is type-clean; it says this check did not run." -
Cache-dir hardening. One directory per attempt under the system temp root, created
0700, refused unless it's ours alone, reclaimed by owner-pid liveness.os.makedirs(exist_ok=True)follows a symlink to a directory and returns happily; theos.lstatis what catches it. Forcing the predicate true turns five tests red.
Eleven paths in that hook return silence. Exactly one speaks.
PR #229 — measuring before building (merged)
Phase 3 of #114 was supposed to be "add a symbol server so the coder can navigate instead of reading whole files." I read it as a gate, not a feature: measure whether navigation would actually pay before building anything.
So scripts/navigation_value.py measures what share of the coder's file-read mass sits in reads a definition / references / hover call could have answered, net of what the symbol answer would itself cost — against a pre-registered 15% threshold, chosen before the number was known so it couldn't be tuned to the answer.
First run said PROCEED at 27.6%. It was wrong. .md was the largest extension in the corpus by read mass, and no symbol call answers a question about a README. With a closed allowlist of languages a symbol server actually serves, the same corpus reads 10.4% / 13.7% and the verdict inverts. One constant flipped the phase gate, so a fixture pair identical except for the file extension now pins it.
Then the maintainer found the deeper version of the same bug. My SYMBOL_QUERY_TOOLS was {Grep, Search}. no_human's coder never emits either — it searches through Bash. From the fleet database:
Bash 115,776 | Read 35,557 | Edit 18,425 | Write 3,010
Grep 0 | Glob 0 | Search 0
Bash calls containing grep/rg/ag/ack: 50,459
So the class my script called its strongest signal was structurally empty, and the script happily rendered a confident HALT on evidence where its own strongest signal could not exist.
The fix: extract searches out of shell commands per binary (option tables per binary, because -r takes no value in grep and is --replace in ripgrep), and make the zero fail closed — NoSearchChannel refuses a corpus with reads and no searches, because an absent channel is a gap in the instrument, never evidence about the agent.
Against the full fleet database, after the fix: symbol_lookup went from 0 to 15.6% of read mass, and the verdict inverted from HALT to PROCEED. The class that was structurally empty became the largest single contributor.
And the output still carries a caution, because the result got less robust, not more: re-deciding at 6,000 chars gives PROCEED, at 24,000 gives HALT.
What I actually learned
1. Understand the constraints before writing code. Both of #164's hard blockers came from reasoning about my module in isolation instead of about where it sits in the orchestrator's integrity window. The architecture is the constraint.
2. A good PR explains behaviour, not implementation. The bodies that survived review are the ones that said what this establishes and what it does not. The "Not covered" and "What I could not verify locally" sections did more work than the design sections.
3. Report the honest number, not the flattering one. 25% of repos, 94% degraded path, a third to two thirds noise. Saying so out loud is what made it landable as advisory evidence.
4. Tests are part of the product. Every guard I added carries a positive control: red on its own defect, green restored. Twenty-four single-line mutations in #229; twenty-three caught, and the one survivor documented as equivalent code rather than quietly ignored.
5. Measure instead of arguing. Three rounds of hard review, and the finding I disagreed with least was the one I'd have lost by arguing. Re-measuring a number the maintainer gave me — three ways, all agreeing — turned out to matter, because his basis had already moved.
6. More automation is not more reliability. Adding an automated type-evidence signal introduced a way to destroy a real review verdict. Every gate you add is a surface.
7. Human approval is not a bottleneck. It's the trust boundary. It's the one place where responsibility actually lands on a person, and it's cheap compared to everything downstream of a bad merge.
What caught my attention in v0.2.4
Release: https://github.com/no-human-ai/no_human/releases/tag/v0.2.4
Approve-and-merge, fixed — a lesson about where "it works" is measured
In 0.2.1–0.2.3 the packaged desktop app could open pull requests but not land them. The merge gate shelled out through the frozen binary as if it were a Python interpreter, so every nh approve and every board Approve failed at the test step. The gate now resolves a real interpreter and decodes output as UTF-8, end to end.
Why it matters: the human decision point is the product's most important surface. A pipeline that opens PRs but can't land them isn't 90% working; it's broken at the only step that requires a person.
Developer impact: if you're on a packaged 0.2.x build, upgrade before you evaluate the loop — you were evaluating a severed approval path.
The review gate is reusable outside a running server
You can now run it from a session as a plugin skill, or on a pull request in your own repository as a GitHub Action — and a fork's pull request is skipped before any credential is read.
Why it matters: this is the change that most affects teams rather than individuals. The verification half stops being tied to "I have the daemon running locally" and becomes something CI can invoke on a human-authored PR.
Developer impact: the evidence-based review gate becomes usable on code the agent never touched. That's a meaningfully different product.
Example: run it on PRs into a shared branch and let the tamper/repro/reviewer output land as a checklist beside your existing CI.
Windows fixes, which are a category and not a footnote
A task whose title or output contains Hebrew, Cyrillic or Japanese text no longer crashes at commit and strands the work. A .env saved with Windows line endings no longer reads as empty and silently hides your credential. The coder no longer loses all codebase context from a drive-letter path.
Why it matters: every one of these is an invisible failure. An empty credential file that reads as absent, or a coder silently running without repo context, produces a bad outcome with no error. Those are the same failure class the verification gates exist for, at the infrastructure layer.
Also in this release
- The running version is shown in About from a single source.
-
nh task add --followsrecords that one task supersedes another. - Signing, stated plainly: macOS is signed, notarized and stapled. Windows is NOT code-signed — the artifacts carry
-UNSIGNED, SmartScreen will warn, and you should check againstSHA256SUMS-windows.txt. Linux ships a.deband an AppImage with checksums. - The release publishes its own known issues, including a residual Windows non-UTF-8 codepage read path due in 0.2.5.
A release that publishes its own known issues is a small signal, but it's the same signal as the rest of the project.
Try it yourself
You need a Claude credential and the Claude Code CLI, whichever way you install — the backend shells out to that CLI for every task.
# prerequisites
npm install -g @anthropic-ai/claude-code
claude setup-token
# install (CLI + board)
uv tool install no-human # or: pipx install no-human
# initialise, then prove the install is real
nh init && nh doctor
# run
nh start # board + worker on 127.0.0.1:8420
nh task add https://github.com/org/repo/issues/42 --repo ~/git/repo
nh status # needs-you / working / waiting / done
nh review <id> # the reviewer's evidence checklist
nh diff <id> # the diff it wants to ship
nh approve <id> # your approval squash-lands the PR
nh reject <id> --reason "..." # send it back with feedback
From source, you also need Python 3.12+, uv, git, and Node with npm — the board is a separate npm run build, and a source checkout ships no web/dist.
What to expect: a task appears on the board, gets a plan, gets implemented, runs your tests, goes through the gates, and stops as an open PR in awaiting_approval. Run nh review <id> before nh diff <id> — reading the evidence checklist before the diff is the habit the whole design is trying to build.
Already inside an agent? There's an MCP server — a stdio bridge on the official Python MCP SDK — exposing exactly two tools, task_add and task_status. It talks to your own no_human at 127.0.0.1:8420 and nothing else.
nh mcp-serve
{
"mcpServers": {
"no_human": { "command": "nh", "args": ["mcp-serve"] }
}
}
For Claude Code, the repo is its own plugin marketplace:
/plugin marketplace add no-human-ai/no_human
/plugin install no-human@no-human-ai
Docs worth reading before you rely on any of it: verification.md (the gates and their limits) and security.md.
When this approach makes sense
The project is explicit that ambitious tasks are not the target. It's aimed at well-scoped work: bugfixes, test gaps, small features, investigations. A vague ticket produces an escalation, and that's the intended behaviour rather than a workaround.
Good fits:
- Bug fixes with a reproducible failure — the repro gate is doing real work here
- Test-gap and coverage tickets
- Repetitive maintenance across a repo
- Investigations where an honest "here's what I found and here's what I can't determine" is a valid deliverable
- Teams who want to experiment with agent workflows without giving up a human merge gate
- CI-integrated workflows, especially now that the review gate runs as an Action
Where a human still has to be in it:
- Architectural change, where the acceptance criteria aren't expressible as tests
- Security-sensitive code — note that a review gate which executes anything repo-authored is a location for exposure, and the project says so in its own docs
- Dependency changes, where the blast radius isn't in the diff
- Anything where the ticket is the actual problem. No gate fixes an incomplete acceptance criterion.
What I'd watch carefully
A balanced read requires saying the uncomfortable parts, and to the project's credit most of these are documented by the project itself.
-
The reviewer is a model, and models miss things. The published confirmation run (2026-08-11,
claude-opus-4-8, 19 seeded defects + 10 controls) measured recall 15/19 (79%) and specificity 7/10. That's a useful gate, not a proof. It also means false positives are routine — plan for triage. - Different models can share blind spots. Independence of session is structural; independence of reasoning is not. Two models trained on overlapping data can agree confidently and wrongly.
- "Different model" is a default, not an invariant. You can configure the reviewer and implementer to the same model. Nothing stops you. That would quietly delete the property the whole design rests on.
-
The read-only reviewer isn't structurally read-only. The docs say this plainly: the edit tools are refused, but
Bashisn't, so a shell redirection could still write a file. The refusal happens at the tool call, in a guard that reads a command line — a cost, not a proof, and the modelled set of spellings is not closed. - The benchmark is self-run and you can't reproduce it. The harness is reusable; the corpus pins to the author's local paths. Success rate moves several points between runs on identical specs, because the coder is non-deterministic.
- No dollar figure is a billed number. Spend is capped in cost-weighted tokens; cache reads were 95.6% of tokens burned in the project's own lifetime measurement. Treat any cost estimate as an estimate.
- Automated tests are a ceiling. Everything here verifies against the tests that exist. A gate can prove a test failed at base and passes now; it cannot prove the test was the right test.
- Language coverage is uneven. Tamper guard reads Python, JS/TS and Java. Repro gate defaults to pytest.
- There is no deploy step. The pipeline ends at an open PR, deliberately.
None of this makes the approach less interesting. It makes it legible — which is the same property the gates are chasing.
The bigger question
The industry's question right now is how much code can AI write? We have an answer: a lot, quickly, and getting faster.
I think the more useful question is:
How much engineering work can AI perform while producing evidence a human can independently check?
Because the second question is the one that determines whether any of this scales past a single motivated developer watching every diff. Volume of generated code is bounded by review capacity. And review capacity isn't bounded by reading speed — it's bounded by how much you have to reconstruct before you can judge.
An agent that hands you a diff makes you reconstruct everything. An agent that hands you a diff, a reproduction that fails at the merge base and passes on the head, a test-integrity delta, a fresh-context reviewer's checklist with verified file:line citations, and a per-rule merge verdict has done part of your job rather than just part of its own.
That's what no_human is actually building, and it's why I found it worth three PRs and three rounds of review that were harder than most code review I've had from humans.
The name is a joke, incidentally. The entire architecture is organised around a human saying yes.
If you're interested in trustworthy AI-assisted development
The project is MIT, runs locally, and is genuinely open to outside work — v0.2.4 shipped contributions from five people outside the core project, each with a CLA entry under contributors/ and their own name on their commits.
Places to start that aren't "write code":
- Read
docs/verification.md. Even if you never install it, the " and what it does not cover" half is the best short document I've read on the limits of automated verification. - Read an open PR thread. The review culture is the product.
- Run it on a small bugfix in a repo you know well, and read
nh review <id>before the diff. - Open an issue about the failure modes your codebase has that these gates wouldn't catch. That's the most useful thing you can contribute to a verification project.
- If you build agent tooling yourself: steal the reproduction gate. It's twenty lines of concept and it's the highest-value idea here.
Project site and docs: https://getnohuman.com/
GitHub: https://github.com/no-human-ai/no_human
Release v0.2.4: https://github.com/no-human-ai/no_human/releases/tag/v0.2.4
If you try it, I'd genuinely like to know which gate fired first on your repo — and whether it was right.
Credits
no_human is built and maintained by Eyal Golan (@eyalgolan). Thanks to him for the invitation to write this up, and for three rounds of review that were more rigorous than the code they were reviewing. Any mistakes in my description of the project are mine; everything I got right about it, I got right because the documentation and the review comments said so plainly, including the parts that weren't flattering.



Top comments (14)
The net-new fingerprint is the part I keep circling back to. Digit-normalising the message so an inserted import doesn't shift every key down a line makes sense, and because it's a multiset the obvious worry (two unrelated diagnostics colliding onto one key) mostly washes out, since the counts stay separate. There's a version that doesn't wash out though: if the base has one error at key K and the head fixes it while a genuinely different diagnostic appears elsewhere that happens to normalise to that same key, the count at K stays at one on both sides and net-new reports zero. A real regression lands in the same breath as a real fix, at the one key least likely to get a second look. Does anything downstream check whether the fixed and new diagnostics are actually the same call site, or is key-count parity the whole signal?
@mihai_leanzero Yeah, this is a really good catch. The key-count parity is not enough to prove that the diagnostic was actually fixed.
The case you described can definitely hide a regression: one diagnostic disappears and another one normalises to the same key, leaving the count unchanged.
The current approach is mainly using the fingerprint as a strong signal, not as proof that two diagnostics are the same. I think the next step would be to carry enough location/context through the fingerprint to distinguish the fixed call site from the new one, while still keeping it stable across harmless line shifts.
This is exactly the kind of edge case I’d want to add as a regression test.
Writing the code is increasingly becoming the easier part. The harder problem is proving that the change actually fixes the intended issue without introducing something else.
I think the verification step needs to be treated as a separate task: tests, edge cases, regression checks, and clear acceptance criteria. An AI-generated fix can look convincing while still failing in less obvious scenarios.
This is probably where AI-assisted development will mature — not just generating code faster, but making the validation process equally systematic.
Well said, @locitra. A fix that "looks correct" and a fix that is "proven correct" can be very different things. I think AI workflows will increasingly need verification loops, not just generation loops. 🚀
I think we're moving toward a world where writing code becomes cheap, but proving correctness becomes the real differentiator. Curious to know—what validation steps do you think should always be mandatory in AI-assisted development? 🤔
Absolutely. I think the key is to treat AI-generated code as a starting point rather than the final result.
My baseline validation loop would be:
The interesting shift is that AI can make implementation much faster, but that makes the verification loop even more important.
Absolutely agree!
I think the next evolution of AI-assisted development is exactly this: from code generation to evidence-backed verification. Great point!
The reproduction gate is the idea I'd steal too, with one scar to add: rerun the base-fail check before trusting it. I once had a fix where the offered test failed at the merge base about every third run. Flaky, not causal, and the single run our check sampled looked like perfect evidence. Running the old tree three times and requiring three failures cost seconds and caught it. The other habit I keep regardless of gates is reading the assertion diff first. A changed assertion is still the cheapest signal there is, and it scans in seconds next to a full reviewer pass. Gates raise the floor. They don't replace that reflex.
@piekwerk
That’s a really good scar to add 😄 The base-fail rerun is easy to overlook, but it makes the evidence much stronger. And I really like the point about reading the assertion diff first — sometimes a few seconds of inspection tells you more than another layer of automation. Gates raise the floor, but that reviewer reflex still matters.
Thanks, and fair point on the layers. One detail on the rerun count, since three was not a principled choice, it was just the smallest number where my one-in-three flake reliably showed up. A one-in-ten flake needs more reruns than anyone will sit through interactively, and that is the real argument for moving the check into CI, where extra runs cost patience you do not have to personally spend. It also composes with your tamper guard nicely, since a base-fail check that the coder can delete is back to the same self-attestation problem.
@piekwerk It makes sense. Three runs is really just a practical signal, not a confidence threshold you can generalise.
I especially like the CI point. Once the flake rate gets low enough, trying to solve it with more interactive reruns becomes pretty awkward. Let CI absorb that cost instead.
And agreed on the tamper guard. The moment the agent can decide which evidence gets checked, the whole thing starts drifting back toward self-attestation. That separation between generating the fix and validating the evidence is probably the more important boundary.
Fair, and I'd keep it at "practical signal" too. What pushed it to three for us was one flaky browser run that passed on retry two: with two runs a coin-flip masquerades as evidence, three makes one flake visible. But the number only means something while the flake rate stays well under one-in-three, and once CI owns the loop, reruns are cheap enough that the exact count stops mattering much.
The one item on your list where the mechanism and the question have different shapes is the third: "Did the test count go down?"
testing/tamper_guard.pydecidestamperedfrom aggregate totals (ta < tb or aa < ab or sa_cmp > sb_cmp or ...) plus an unconditional deleted-file rule. The per-file loop records drops independently inreasons, andreasonsis never consulted for the verdict. The module's own comment block names the shape — "the guard named the tamper and passed it" — and two instances of it are now closed: the deleted-file case (promoted into the verdict asbool(deleted)) and the two path shapes whose benchmark fixtures were diluting the aggregate. What got closed is those instances, not the rule.Reproduce it with your own test's bytes. Take
test_fixture_snapshots_do_not_dilute_a_real_reductionand change one thing — the padding file's path, from undereval/reviewer_recall/cases/control-gate-excerpts/base/totests/test_snapshot.py:Same bytes as your test, same guard, same commit (
5f99b7af) — only the excluded path shape is gone, and the verdict flips to clean with the reduction still named in its own reasons list. A branch that adds a test file anywhere outside those two shapes can delete assertions from an existing suite and be reported clean; the padding only has to exceed the deletion, and padding is the one resource an agent has in unlimited supply.The fix is what the per-file loop already computes. Decide from it:
tampered = bool(deleted) or bool(reasons). I ran that rule against the three honest-refactor control pairs already intests/test_tamper_guard.py— thedescribe(→test.describe(rename, the.test(helper swap, the assertions-added case — andreasonsis empty in all three, so none of them changes verdict. A per-file rule does fire on an honest relocation (a test moved between files), which is the trade your deleted-file rule already took explicitly, with the justification written next to it: the guard fires, and a human justifies it.The reason this one is worth more than it looks is in your own argument — "if your verification mechanism is inside the agent's write surface, it is not a verification mechanism." A rule that decides on a total is still inside the write surface, because the total is something the writer can add to. The per-file facts are the part that cannot be padded.
@howcani_howcani_77e786a89
The aggregate-versus-file distinction is the key point for me.
An aggregate check looks at the overall test and assertion totals. That means a real reduction in one file can potentially be hidden by adding enough tests somewhere else.
The per-file check answers a different question: did any existing test file actually lose tests or assertions? That evidence is much harder to mask with unrelated additions.
So the issue isn't that the aggregate numbers are useless. They are useful as a high-level signal, but they shouldn't be the only thing deciding whether a reduction happened. The per-file evidence should have a direct path into the verdict.
The test relocation trade-off is also important. A per-file rule can flag legitimate moves, but that's a review case rather than something I'd want to hide by making the detection weaker.
A fix is only worth anything if the check that proves it runs without the agent anywhere in the loop, because an agent asked to evidence its own fix will happily produce a passing test written against the new behaviour. A check written after the fix tells you almost nothing, because it was written to agree with whatever the fix already does. Writing that check first, while the bug is still reproducing, is the only moment it costs nothing. Strip the agent out of the verification path and you find out quickly whether the fix was real or just plausible.