This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content
What I Built
My ledger says three outside engineers found the same defect. Ask the agent how many locatable receipts support that row and it says one.
Finding B8 has 1 locatable comment (
3ee98by Pushpendra). Although three outside finders
are recorded, the receipts for Vinh Nguyen and quashudev were not found in the comment trees
searched.
It does not turn three recorded names into three verified receipts. The dataset stores finders and
receipts as different things, and the agent is not permitted to add them together.
It does that over a public Sanity record of 120 documents: 74 articles, 14 findings, 10 people,
3 patches, and 19 claims. The claim documents separately store asOf, status and
expiryStatus, rather than asking the model to reconstruct those distinctions from prose.
Three things it will tell you about its own author:
- the fix for finding B1 is not merged into
origin/main— it reportsinMain: false - of 14 findings, 12 were raised by the 8 outside engineers and 2 are internal (one raised by my coding agent,
recorded as
Ka'el, and one by an internal audit). Two are implemented; twelve are not - it cannot see what happened to that branch after the record was built, and says so
Demo
https://ask-the-record.vercel.app — no login, no user-supplied key, five buttons.
Measured answers on this deployment ran 13.8–33.8 seconds across the five questions — the raw
timings are in evidence/web-latency-2026-09-21.json. Evidence-bearing
answers must survive a retrieval check: a substantive verdict returned with no successful read is
marked a contract violation. INSUFFICIENT_EVIDENCE is allowed to abstain without pretending a
source was retrieved. Retrieval or auth failures return an error instead of an answer. Every
answer that renders cleanly is required to carry five fields:
ANSWER the claim, stated plainly
SOURCES what it read, and any source URL carried in the record
EVIDENCE DATE the asOf recorded in the document — never today's date
VERDICT STANDING · RETRACTED · SUPERSEDED · UNBUILT · EXPIRED
NO_EXPIRY_SET · INSUFFICIENT_EVIDENCE
UNCERTAINTY what this answer cannot establish. Never empty.
Press the Kubernetes button and it returns INSUFFICIENT_EVIDENCE and no number, because the
Knowledge Base evidence available to that question does not establish a count. It is not claiming
the dataset could never yield one. On the graded run, the agent abstained rather than supplying a
number; the validator does not independently establish that every abstention contains no unsupported claim.
The page also checks its own output. If an answer is missing a required field, has empty
uncertainty, or fails the citation requirement for the instrument that produced it — a URL for
dataset answers, an entry path and the Knowledge Base id for Knowledge Base answers — the page
shows the violation instead of rendering the answer cleanly. It checks citation presence and
syntax. It does not resolve URLs, and it does not yet prove that every emitted citation belongs
to the evidence actually retrieved.
Keys live server-side. The browser never receives one.
Code
https://github.com/keniel13-ui/ask-the-record
studio/ the schema — person · article · finding · patch · claim
harness/ a CLI version of the same agent, and the contract it was built against
web/ the serverless function and page behind the demo
evidence/ breaker verdicts and unedited transcripts
The Python harness and the serverless agent use only the Python standard library. The Studio uses
Sanity's normal React and TypeScript dependencies.
How I Used Sanity
The schema makes the distinctions representable; the endpoint instructions require the agent to
keep them separate; the harness independently enforces retrieval, required output fields, citation
syntax and named-object guards around the answer. It does not yet re-verify the returned values against the
document field by field — the instructions carry that, not the validator.
One honest wrinkle in my own design: the response VERDICT is a single enum that mixes claim state,
finding state, expiry metadata and retrieval outcome. That is a presentation-level classification.
The underlying status and expiryStatus stay separate in the dataset, and the graded claim lookup
reported both separately in its answer text — but the validator does not yet guarantee that
separation for every future answer, and a stricter contract would split the output the way the
schema does.
Three schema choices do the work:
1. status and expiryStatus are separate. A claim can be standing and no_expiry_set
at the same time. no_expiry_set means nobody dated it — not "permanently true," not "expired."
Collapsing those two facts into one is how a record quietly rots, so the schema represents them as
separate fields instead of forcing one value to stand for both.
2. A finding has commentOn and writtenUpIn. Where someone raised a defect and where I wrote
it up are different articles. pm25coder found B1 in the comments of one article; it was written up
in another. One field would have glued the wrong attribution to the wrong post.
3. patch.inMain is a boolean. Public is not merged. All three recorded patches are pushed and
none is in main. Asked about B1, the agent reports inMain: false rather than inferring merge
state from the branch being public.
Two instruments, because one couldn't do it
I built this on a Knowledge Base first. It handled the four prose-routed frozen questions in the
graded run. But I could not reliably recover a specific claim's status and expiry from the indexed
representation, and the measurement shows why:
all 24 Knowledge Base entries = 190,503 characters
'claim-ledger-population' = 1 occurrence — a label in a Sources list
its values = present, but not bound to that identifier
The values existed in the indexed prose but were not bound to claim-ledger-population. A prompt
cannot make that binding reliable when retrieval did not preserve it, so I routed that question
shape to GROQ — which is what Sanity recommends when the schema already says where to look.
I did not grade my own work here. Independent review means separate, owner-assigned AI reviewer
sessions that did not implement the candidate — not outside human testers. The first live graded
run they carried out exposed the wrong-object failure on exactly that question. Synthetic breaker
controls then found more: failed retrieval could reach the model, unsupported answers could exit
zero, parts of the output contract were not enforced, and the contracted tool-discovery step was
missing. The VERDICT.md in evidence/ is a
BLOCK and it stays there.
The fix was a second Context endpoint serving GROQ over the same dataset:
*[_id=="claim-ledger-population"][0]
→ status "standing" · expiryStatus "no_expiry_set" · asOf "2026-09-11"
sourceUrl "https://dev.to/kenielzep97/my-harness-used-one-label-for-three-different-failures-2gc3"
One document, its fields, and its stored sourceUrl.
So the agent routes by question shape. Four of the five frozen questions — B1 attribution, B1
merge state, B8 receipts and Kubernetes — stay on the Knowledge Base. Only the exact claim-field
lookup routes to GROQ.
The router itself is blunter than that description: any question containing a claim-* token goes
to the dataset endpoint, which is broader than the rule I specified. The breaker preserved that as an open
remainder rather than letting me describe the abstraction as if it were the code.
By default, a Sanity Context endpoint's mode is derived from its configured sources: an endpoint
with a dataset source serves GROQ mode, and Knowledge Base sources on that endpoint are ignored. So
two endpoints, exactly one instrument per question.
self-correcting-systems initial_context, knowledge_base_read
self-correcting-systems-data initial_context, groq_query, schema_explorer, array_field_reader
Custom instructions on each endpoint carry the rules — report receipts not claims, keep status and
expiry apart, bound anything about mutable state to the snapshot date. They arrive through
initial_context. The harness fetches that context before letting the model read a Knowledge Base
entry or query the dataset, so the endpoint's instructions are in context before any evidence is
retrieved.
What it does not establish
The current corrected CLI harness held all five frozen questions on one independent v8 graded run
with gemini-3.6-flash. That is not general reliability. The earlier BLOCK used gemini-2.5-flash;
I am not claiming the same model failed and then passed.
One more limitation is preserved in the breaker verdict rather than smoothed over. The graded v6
candidate enforced the Knowledge Base citation rule incorrectly — it accepted an entry path or
the Knowledge Base id where the contract required both. The five live answers happened to supply
both, so their content held, but the guard was weaker than specified. Version 7 corrected that to
an AND and received its own independent run. A later review found that an unparseable VERDICT
could bypass the verdict-dependent checks. Version 8 rejects those malformed values and received
a new independent run: five of five, exit 0. Its CLI hash matches harness/ask.py in the repository.
The v6 and v7 transcripts remain evidence for their respective versions.
All four graded runs — the original BLOCK, v6, v7 and v8 — are in evidence/, unedited.
No earlier transcript was replaced to conceal its result.
Agent Session / Evidence Trail
I kept the failed run instead of replacing it:
- the original BLOCK and its unedited transcript
- v6 — five content-holding answers, weaker citation guard
- v7 — the AND fix, five of five, exit 0
-
v8 — after a review seat found that an unparseable
VERDICTskipped every check gated on it; five of five on the repaired file - the latency receipt — five anonymous live requests and their measured timings
The historical verdicts remain unedited. GET /api/ask reports the sha256 of the deployed function,
so you can check which source is actually serving rather than trusting that a push landed.
Sanity Project Details
Project ID u58x3mt0
Dataset production (public)
Knowledge Base kbjnxAgyAimV
Studio https://self-correcting-systems.sanity.studio/
Query the record yourself, no credentials:
https://u58x3mt0.api.sanity.io/v2025-08-15/data/query/production?query=count(*)
*[_type=="finding" && status=="unbuilt"]{code, title, "who": foundBy[]->handle}
No login is required for the demo or the dataset.
I'd rather ship an agent that says two of fourteen are implemented, one of three B8 receipts is
locatable, and this record cannot see past its snapshot than one that turns every stored assertion
into certainty.
Top comments (10)
The distinction between having a receipt and proving a claim is an important next layer here. An agent can return a valid source URL and still produce a conclusion that the retrieved evidence doesn't actually support. That makes evidence validation more interesting than citation validation: the system needs to establish not just “did we retrieve something?” but “does this retrieved record justify this specific claim?” I also like the explicit uncertainty field because it makes that boundary visible to the caller. As agents become more evidence-driven, I think the strongest contract will be one where every substantive claim has a traceable evidence path, while unsupported claims are forced into an explicit abstention state rather than being filled in from model inference.
You've named the gap correctly and I'd rather say it's open than imply it isn't.
What the validator does today: checks a citation is present and syntactically well formed.
It does not resolve the URL, and it cannot tell whether the retrieved record supports the
specific claim next to it. Citation validation, not evidence validation, exactly as you
split it. An answer can carry a real URL to a real document and draw a conclusion that
document doesn't support, and the contract passes it.
There is one rung between your two, though, and it's the only part I'd defend. If the
question names an identifier, that identifier has to appear in the text retrieval actually
returned, or the answer is marked a violation. So it can't cite a real record while having
read a different one. That's weaker than justification and stronger than "did we retrieve
something."
On your last point, that part exists. A substantive verdict with zero successful reads is
a contract violation, and INSUFFICIENT_EVIDENCE is the only verdict exempt from the
evidence checks. Abstention is the cheap path on purpose.
What I don't have is the justification check. My honest read is the claim has to be
decomposed into checkable parts first, and most claims aren't shaped that way.
That makes the boundary much clearer. The identifier check is a useful middle layer because it proves the cited record is actually the one the agent retrieved, even if it doesn't yet prove the conclusion is justified. I think the claim-decomposition problem is the interesting next step: once a claim can be broken into smaller checkable assertions, the validator can evaluate evidence at that level instead of treating the whole answer as one unit. That could also make partial support visible, rather than forcing the system into an all-or-nothing evidence verdict.
Partial support is the part I want most. Right now every verdict covers the whole answer. It's clean, a contract violation, insufficient evidence, or a retrieval error, so an answer with three solid claims and one made up one still gets one verdict for all four. Breaking it into claims first would let it point at the one that isn't supported instead of just saying something's off. Haven't built that yet, but yeah, that's the next layer.
Kept the BLOCK verdict in the evidence folder — that alone made me trust the writeup more than a clean green run. The finders-vs-receipts split is the kind of distinction a keyword search can’t fake.
Curious how you’re thinking about the next contract layer (does the retrieved record actually justify the claim, not just cite a URL).
The BLOCK stays because deleting it would have been the more interesting failure.
The gap you're pointing at is real and still open: a valid URL next to an unsupported
conclusion passes my current checks, because I verify the citation exists, not that it
justifies anything.
I don't yet know how to make that inference checkable without first decomposing the
claim. "One locatable comment supports B8" is checkable because it reduces to a count
over a named field. Most claims aren't shaped like that.
If you get further on that than I have, I'd genuinely like to hear it.
Great read! I've been working on something similar, from the other side: a harness where coding agents build and review each other's work, and nothing counts as done until a gate the agent can't write says so.
when a claim is standing with no_expiry_set, does EVIDENCE DATE ever push the VERDICT toward uncertain on its own, or is staleness left to the reader?
Straight answer: no. Staleness is left entirely to the reader.
I checked before replying rather than from memory. Zero date comparisons in the harness.
EVIDENCE DATE is an instruction to copy the record's asOf and never use today, and the
validator only checks that the field is present. Standing plus no_expiry_set with an asOf
from March is treated identically to one from yesterday.
On your gate the agent can't write, there are two in there and they're the only reason I
trust any of it. The read counter and the retrieved text are appended solely from the
endpoint's response, never from model output. So "answered with a verdict but read zero
entries" and "the question names B8 but B8 is absent from retrieved text" are both
decided outside anything the model can influence. Everything else in that validator is
string checking and I'd call it decoration by comparison.
Which is why the date thing bothers me. It's the one field the model writes and nothing
verifies, so it's exactly the shape of gate I claim not to trust.
My current thinking is that no_expiry_set should be loud rather than neutral. It reads
like a shrug when it should read as "nobody ever committed to when this stops being true."
How do you stop the reviewing agent from being talked into a pass?
Some comments may only be visible to logged-in visitors. Sign in to view all comments.