Jev labels the line. Code makes the call.
TL;DR: README Clew 2 checks whether a public GitHub README matches its code, for JavaScript and TypeScr...
For further actions, you may consider blocking this person and/or reporting abuse
This is an incredible example of architecture-first AI engineering. Starting with a strict constraint—a deterministic decision model like Jev that 'can't write a sentence'—and using it to solve the exact architectural blind spots of a traditional LLM pipeline is textbook defensive design.
Your submission for the 'What Would Jev Do?' hackathon highlights some brilliant patterns for production-grade agent validation:
Thank you for reading this so closely, and for taking the time to share ideas!
I love the thinking behind a second check with Haiku. I went a slightly different way here. My one rule for this build was that only code decides whether a line is true, never a model. So when Jev isn’t sure, the line goes into “can’t tell.” It’s a little cautious, but I’d rather say “can’t tell” than accidentally call a correct README wrong. But it’s got me thinking: how would you add a second check like that without letting the model make the final call?
Your idea of pointing each label at one part of the code is close to how it works now. A line about a package gets checked against the package list, a command against the scripts, and so on. Nice to know we landed in the same place.
One fun thing I learned along the way: Jev can give the same line a different label on a second run. That’s part of why I only send the lines it’s very sure about to be checked.
Thanks again!
That rule makes total sense—keeping the final decision in deterministic code is definitely the safest way to avoid false positives. To add a model like Haiku without giving it the final call, you could use it strictly as a "confidence booster" to re-evaluate the "can't tell" lines, but still run those suggestions through your hardcoded validation logic before marking them true.
That’s a clever way to do it, and I like that the code still gets the final say. The one snag for me is that I also made a rule that Haiku never reads the README itself. It only sees the results. That way, if someone hides instructions in a README, the summary model never sees them.
But your idea got me thinking about a version that fits both rules: asking Jev a second time, in a different way, and only checking the line if both answers agree. I don’t know yet if it would work. If Jev misreads a line once, it might misread it the same way twice. It would need a real test before I’d trust it. Thanks for pushing on this, it’s a fun one to think about!
Isolating the summary model from the raw README to completely mitigate hidden prompt injection risks is a brilliant architectural boundary. Relying on multi-turn consensus filtering with Jev to cross-verify the results is exactly how we keep multi-agent LLM pipelines deterministic and robust. Prototyping some dev tools around this exact pipeline validation blueprint right now.
I appreciate it! I’m glad the idea of asking Jev twice landed with you too. I haven’t tried it yet, so for now it’s just a fun one to think about. Keeping Haiku away from the README helps a lot with sneaky instructions, though Jev still reads the README to label it. That’s why I let code make the final call. Good luck with your tools, I’d love to see what you build!
The 32-line held-out evaluation is a useful starting point, but I think the regression suite is where this approach will prove its long-term value. One edge case worth testing is a README command that exists in a workspace package but becomes invalid after a dependency or script changes in a different workspace. The verifier needs to catch the actual drift without turning unrelated package changes into false contradictions.
I’d be interested in tracking precision separately for each claim type, especially commands, dependencies, and environment variables. An aggregate zero-false-contradiction result is encouraging, but per-category results would make it much easier to see where the verifier is genuinely reliable and where it still needs more coverage.
This is so useful, thank you. You’re right on both.
32 lines is a small test, and my README says so. The saved real-world tests are where the proof will build up over time. Two of them came straight from real mistakes: my own old repo, and a repo whose examples folder fooled it.
Your edge case is a good one: a command in one package breaks because of a change somewhere else, and the tool has to catch that without blaming lines that weren’t touched. I haven’t tested that exact case yet, so it’s going on my list.
I love the idea of tracking accuracy separately for commands, packages and settings. With only 32 test lines, the split would be too thin to mean much right now, so I’ll need a bigger test set first. I’d rather show that honestly than show three numbers that look better than they are.
That distinction between catching real drift and avoiding false contradictions is the tricky part. In a monorepo, for example, a shared package might change its exports or upgrade a dependency, breaking a documented command in a downstream workspace even though the README and that workspace’s own
package.jsonhaven’t changed.I’d be curious to see whether Clew could eventually use the workspace dependency graph to identify which claims need rechecking after a change, rather than treating every claim as equally affected. That could make the regression suite more useful as repositories grow, while keeping unrelated changes from generating noise.
And I agree with holding off on per-category accuracy numbers until there’s enough data behind them. A metric is only useful if the test set can actually support the conclusion.
That example makes it click for me. The README didn’t change and the package didn’t change, but something it depends on did. That’s the kind of drift that’s hardest to see.
Clew 2 can’t catch that yet. It checks a repo at one moment and doesn’t keep any history, and it doesn’t follow how packages depend on each other. The closest thing it has is the downloadable report, which saves the exact commit and a fingerprint of the README. Two of those, before and after a change, would be the start of what you’re describing.
I really like the idea of only rechecking the lines a change could touch. That would keep the noise down as repos grow.
And I agree on the metrics. A number is only as good as the test set behind it.
I love that the first thing it got wrong was your own repo. That's the most honest test any tool can have. And the fact that you don't write the code yourself, you direct, validate, and decide, is exactly how I think most of us will work in a few years. Respect for shipping this in 5 days. And yeah thanks for sharing..
Thank you! Scanning my own old repo first was half honesty and half nerves. Better it embarrassed me than a judge finding it. And yes, that’s how I build: I decide what it should do and what’s true enough to ship, and the agents write the code. Glad it landed.
The failure mode I'd watch for in this architecture isn't a weak judge — it's judge erosion.
Once the generator and the judge are separate, every disagreement between them creates pressure in the same direction: the judge said no, and fixing the generator (or the content) is slower than relaxing the judge. One exception is reasonable. Ten exceptions later, the judge still runs, but it's ceremonial — it says yes to everything because saying no got expensive.
The rule that's worked for me on the static sites I build: the judge only ever gets stricter, and loosening it costs more than fixing the page. If a label keeps failing, the default assumption is the page is wrong, not the check. I also watch how often a single check trips — if it's constant, that's a signal to redesign the check or the template on purpose, never to quietly raise the threshold.
Separation is the right call. Keeping the judge honest after the separation is the second project, and it's the one that quietly decides whether the first one still means anything.
“Judge erosion” is a great name for it, and I can see how easily it would happen.
I had a version of this during the build. My tool’s first real test was my own old repo, and it called three correct lines wrong. In that case, the check really was the problem, so I changed it. But I tried to only ever change it in one direction: when the tool isn’t sure, it says “can’t tell” instead of “wrong.” The bar for sending a line to a check stayed where I set it.
Your rule about watching how often one check trips is a good one. If the same check keeps failing, that’s a design question to answer on purpose, not a number to quietly nudge. Thanks for this.
Fascinating deep dive into AI workflows. Grounding outputs and evaluating latency budgets will be key as more agentic systems reach production.
Thanks! Grounding was the whole design for me. Jev only labels what kind of claim a line is, and a code check against the actual repo decides if it’s true. Even the summary model only sees the checked results, never the README itself, so it can’t invent anything about it. Curious how you’re grounding outputs in your own work?
Thấy approach "Jev labels → Code judges → Haiku explains" khá thú vị — tách biệt rõ ràng ba giai đoạn: gán nhãn dữ liệu, đánh giá logic, và sinh ngôn ngữ tự nhiên. Cái hay là việc dùng AdaL (Adaptive Labeling?) để giảm thiểu human-in-the-loop ở bước labeling, nhưng vẫn giữ Code làm gatekeeper cuối cùng.
Hỏi chút về pipeline: Haiku sinh giải thích dựa trên output của Code (pass/fail + reasoning), hay nó có truy cập context nguyên bản (diff, file tree, issue linked)? Vì nếu chỉ feed binary result + short reason thì dễ bị hallucinate hoặc generic. Thấy nhiều project gặp vấn đề này khi compress context quá gọn.
Còn về performance: AdaL có feedback loop tự động retrain khi Code reject label của Jev không? Hoặc hiện tại vẫn human review các false positive/negative? (site: labagent .tech)