Deeply understand the task at hand
The biggest mistake teams make with agents is automating a process they don’t understand. Before writ...
For further actions, you may consider blocking this person and/or reporting abuse
The "data is the bottleneck" section matches our experience painfully well. Clean synthetic data makes an agent look brilliant offline and then fall over the first time it meets a real customer's messy Excel export — the mess genuinely is the signal, and most teams under-invest there.
Strong agreement on rubrics over pass/fail too. Moving to multi-criterion scoring was what finally let us see "4 of 6, and here are the two it missed" instead of a meaningless red X.
One trap that bit us: an LLM judge grading only the final output will happily pass a right-looking answer reached by wrong reasoning — correct number, broken trajectory. We ended up adding assertions on the trajectory itself, not just the output. Did you run into that with your worlds? And how do you keep the domain-expert rubric from going stale as the product shifts — ours drifted faster than we expected.
The roll-forward worlds are a useful way to test whether memory helps beyond the first task. I would also grade provenance: can the agent distinguish a reusable workflow lesson from a period-specific value it must re-fetch? That turns “fresh numbers” into an observable failure mode rather than a guideline.
The skill ablation result (eight skills trimmed to four, some of them hurting) is the part I'd copy. One thing to check if your harness hands work to subagents: per the Claude Code subagents docs, each subagent starts with a fresh, isolated context window and does not see the conversation history or the skills already invoked in the parent, except a fork, which inherits the parent conversation. So ablating a skill on the main agent says nothing about what a subagent does with it.
The eight-to-four skill ablation is a useful reminder that extra instructions can hurt. I'd keep the per-world score changes beside the 2.8-point mean improvement, and set a minimum for any criterion the domain expert considers essential. Otherwise a gain in several worlds could hide a regression in one. The same table would make the 61% compression result easier to judge: unchanged average quality is reassuring, but I'd also want to know whether the difficult cases stayed intact.
The roll-forward worlds make the memory test much more convincing. A useful extra case might keep the business familiar but change one workflow rule next month, then check whether the agent notices the new evidence instead of applying last month’s successful procedure automatically.
The 'worlds' idea is the part I'd steal: a simulated business with full books and planted mess. One thing I'd add is generating the world from declared targets (a known accrual total, a variance you planted on purpose), so you get an answer key for free and some criteria can be checked exactly instead of by the LLM judge. How do you build the worlds today, by hand or generated? I build Misata, a synthetic data generator that does the declared-targets part for multi-table data, so I'm curious where the hand-work still is.
the 'remember workflow, forget numbers' finding is the clearest result in here — memory encoding reusable reasoning patterns survives rollover, memory that embeds last period's figures degrades the next run.
what's subtle is that both look identical at extraction time. you need that rollover fixture to expose which class you're actually storing, and most teams skip it because 'time passing' is hard to simulate.
we added assertions on tool call ordering as a proxy: does the agent read before writing, investigate before correcting? an LLM judge on final output alone will pass correct answer with broken trajectory.
do your rubric criteria touch trajectory at all, or stay on output quality?
The skill ablation result is what I keep rereading: going from eight skills to four, gaining 2.8 points, and finding that several skills actively hurt. For months I kept adding context to an agent on the assumption that more could only help, and I never ran a no-skill baseline. When a skill helped in one world and hurt in another, how did you decide whether to keep it?
The worlds approach for synthetic data is the detail most teams skip. Clean test data is a trap — the agent aces it in dev and falls over on the first real invoice with a weird format or missing field.
Breaking into categories before writing cases is also doing real work. It forces you to have an actual opinion about what the agent should be good at rather than writing tests that confirm what you already built.
Curious how you handle partial correctness in grading. Like the agent finds the 500 dollar discrepancy but attributes it to the wrong bill. Full fail, half credit, or something more nuanced?
The messy synthetic data point is the one I'd underline. Evals built on clean data look good offline and die on real users, and a rubric that shows 4 of 6 tells you exactly where the agent is weak.
What specific metrics are used in the end‑to‑end evaluation framework?