Generation got cheap. Trustworthy review did not.
So we add reviewers. More eyes on the PR, more verifiers in the gate, a panel of LLM judges instead of one. The assumption underneath is that each additional reviewer adds independent evidence.
That assumption is measurable. I measured it, and it did not hold.
What IDKMesh is
IDKMesh is an open-source research project (Apache-2.0, Python 3.11+) asking how humans, AI agents, tools, and heterogeneous compute can coordinate on uncertain goals and turn proposals into verified useful work.
It is a research preview, not production software, and the repository is deliberately conservative about what it claims: every number below traces to a committed experiment record you can re-run.
The thesis in one line: reviewer count is not independent evidence count.
How it works
The coordination model treats generation as cheap and verification as the scarce resource:
- Work arrives as a bounded Work Unit that must declare its security bounds.
- Replaceable workers attempt it. Multiple attempts are expected; candidates are disposable.
- A separate, verifier-owned evaluation plan judges those candidates. Worker completion is not acceptance.
- Results become reproducible evidence - result manifests, verification results, provenance digests.
- Integration is an explicit decision gated on that evidence, not on a green impression.
The bundled demo makes the boundary concrete: it runs the real validators against committed fixtures and deliberately rejects four invalid ones - including a task with no security contract, and a worker result that accepts itself.
Measuring a gate
idkmesh gate-audit takes verdicts you have already collected and reports what the panel is actually worth. Here is the bundled example:
{
"schema": "gate-audit-report-v0.1",
"gate_id": "example-five-verifier-gate",
"evidence_class": "synthetic",
"inputs": { "candidates": 15, "verifiers": 5, "known_good": 8, "known_bad": 4 },
"panel": {
"nominal_votes": 5,
"mean_verifier_accuracy": 0.8,
"mean_pairwise_error_correlation": 0.0916,
"effective_votes": 1.6944,
"heuristic_n_eff": 3.6588
},
"probes": { "total": 3, "breached": 2, "breach_rate": 0.667 },
"warnings": ["2/3 seeded known-bad probes were accepted by the panel"]
}
Read the third key before the interesting one: "evidence_class": "synthetic". Those five reviewers are invented, and 1.69 is a demonstration of the arithmetic, not a measurement of anything real. The project keeps a hard line between synthetic demonstration and observed evidence - so here is the observed side.
The observed result
Experiment E017 built a panel where every verifier is a program, not a model: a partial test oracle that draws inputs from one named region of a problem's input domain (tiny, small, large, extreme, duplicate) and accepts a candidate only if it matches a reference implementation on all of them.
5 regions x 5 seeds = 25 verifiers, run over a 72-candidate corpus whose ground truth comes from executing hidden tests. Every verifier's Youden J is significantly positive after Bonferroni correction - mean accuracy 0.7956. So the panel genuinely works, individually.
Error correlation between them:
same region (declared dependent) mean rho = +0.8924
diff region (declared independent) mean rho = +0.5263
all pairs mean rho = +0.5873
Verifiers that share no declared attribute still share 53% of their errors. A metadata group boundary is not an independence boundary.
Then the part that matters:
25-verifier majority error : 0.2083
single verifier : 0.2044
measured effective size : 1.00 (of 25 nominal)
N/(1+(N-1)rho) heuristic : 1.66
The panel was worth no more than one of its members. Twenty-five nominal votes, an effective size of 1.00, and the standard correlation correction still overstated it by 1.66x.
The whole measurement ran in about 5 seconds on one laptop.
Why the usual correction doesn't rescue this
The familiar fix is to discount for correlation with N_eff = N/(1+(N-1)rho). Experiment E015 tested that heuristic across a parameter grid and found it exact at rho=0 and rho=1, wrong in between, and - importantly - the sign of its error is not fixed.
It is conservative where verifiers are weak, which is where conservatism is cheap. It is optimistic where verifiers are accurate and share modest dependence, which is exactly the regime you care about. At p=0.90, rho=0.125 the measured effective size saturates at 4.60 against a heuristic asymptote of 8.00 - and where an independent 9-verifier panel would deliver balanced error 0.000891, the real panel delivers 0.0125. Fourteen times worse.
What this does not claim
Being precise about scope, because it is easy to over-read:
- No AI review panel has been measured in this repository. E017's verifiers are programs. Whether LLM reviewers exhibit the same correlation structure is an open question - and the one I most want data on.
- It is a research preview. Not production software.
- It is not on PyPI yet, so installation is from a clone.
Try it
git clone https://github.com/MSKazemi/idkmesh && cd idkmesh
python -m venv .venv && source .venv/bin/activate
python -m pip install -r requirements-phase0.txt
python scripts/demo.py
No model account or API key needed.
Repository: https://github.com/MSKazemi/idkmesh
If you have verdict logs from a real review gate - human reviewers, LLM judges, or CI checks - pointing gate-audit at them is the experiment I would most like to see someone else run. Especially if it disagrees with the above.
Top comments (7)
We ran into this last quarter with five judge models, all individually passing calibration, and our consensus gating was basically theater because when one got something wrong the others usually did too. At the time we just noticed the effective coverage seemed much smaller than the panel size and started rotating model families, but we had no measurement to explain why it mattered or convince anyone it was a real problem worth addressing. The 0.5263 is the number I wish I'd had six months ago.
That's exactly the case gate-audit is for — you had the right instinct (rotation helped) but no number to defend it with. If you've still got those judge verdict logs, pointing gate-audit at them would be the first real LLM-panel data point I've seen — E017 is only program verifiers so far. Curious whether it lands near 0.53 or worse.
I don't have the logs in a format gate-audit can ingest, but the number I do have: 94% of decisions were unanimous across the five judges, and majority vote on the 6% disagreements matched my human labels at 71% vs 89% for unanimous cases. When I first saw that gap I assumed it was noise, but it reproduced across two independent labeling passes, which made me think the judges share something structural rather than just tending toward the same token patterns. My rough guess is effective size lands closer to 2 than 5 on your scale, but I would genuinely run it if you share what the log schema looks like.
The self-accepting worker case in the demo is what makes "reviewer count is not independent evidence count" measurable, not just a thesis line. A panel can return effective size 1.00 and still pass a naive audit if all you measure is whether verifiers agree with candidates, because that's agreement with the worker's own completion claim, not evidence beyond it.
The explicit rejection of "accepts itself" is the boundary separating a real gate from a panel agreement score. Without it, the number is just consistency among similar models.
Yes — that's the exact line the demo draws. A gate that only checks "do verifiers agree with the candidate" can hit effective size 1.00 and still look healthy if the candidate is also doing the grading, which is what the self-acceptance fixture exists to catch. "Consistency among similar models" vs. actual evidence is a good way to put it.
The number I'd put on a dashboard is not the panel accuracy — it's your probe result (2/3 seeded known-bad accepted). Individual Youden J looked fine while the panel added nothing, which is basically the case for treating a fixed, versioned probe battery the way you'd treat a regression suite for agents: it catches the drift that per-verifier metrics can't see.
One prediction for LLM reviewers specifically: their error correlation should be worse than your program verifiers', not better — they share training data and tokenization-level blind spots, so systematic errors won't show up in per-verifier accuracy. That puts real LLM panels squarely in the regime where the N/(1+(N-1)rho) heuristic is optimistic.
Would genuinely love to see gate-audit pointed at real LLM-judge verdict logs.
Agreed, and that's my prior too — these program verifiers were picked to be independent-ish (different input regions) and still average rho=0.59; LLM judges sharing training data and tokenization have less reason to be independent, not more. The probe-battery point is exactly right — Youden J alone missed that 2/3 known-bad candidates got through, which per-verifier accuracy can't surface. If you get real judge logs into gate-audit, that's the experiment I most want to see.