I've been building AI agents for a while, and I keep hitting the same wall.
It's not the model hallucinating. It's not the API being slow. It's that the agent works perfectly in my test environment, with my 20-message context, and then fails in production with a 200-message context, a different tool chain, and a user who asks something I didn't anticipate.
The failure modes are different from traditional software:
- The agent doesn't crash. It just makes a slightly worse decision.
- The tests pass. The context window just got too big.
- The demo works. The real user asks a question that's 5% different, and the whole chain breaks.
I wrote about this before ("retry loops are expensive"), but I'm curious about the broader problem. How do you test an agent for the "messy context" it'll see in production, without shipping it to production first?
What does your test suite look like for agents? Do you have a "context window full of noise" test, or is it still mostly happy-path?
Top comments (11)
The spreadsheet screenshot example in the comments is the real failure mode, not the retry loop. A pasted screenshot gets downscaled by the model before any text extraction happens, so messy context from an image is worse than messy context from text, the model is reasoning over a blurrier version of what the user actually saw.
Spot on. The vision encoder usually squashes high-res screenshots down to a fixed resolution before the model even processes it. So a full spreadsheet becomes a blurry thumbnail, and the AI is basically just guessing. Text will always beat pixels for dense data.
On Claude it's not a fixed size, it fits the long edge into 2576px (1568 on older models), so tall captures suffer the most. A wide spreadsheet survives, a long one turns into a thin strip.
Text does cost more though. On a tall page the text version came out around 11,372 tokens against 448 for the image, way more accurate but not cheaper.
That 2576px long-edge limit explains exactly why my massive code files turn into unreadable thumbnails when I screenshot the whole editor.
The token math is wild, though. 11,372 text tokens vs 448 image tokens is exactly why the "screenshot everything" approach feels cheap but silently destroys accuracy on tall pages. Paying the text token premium is definitely worth it for data extraction, but for general UI debugging, 448 tokens is a steal.
Have you experimented with slicing tall screenshots into multiple smaller images (like overlapping crops) to get the best of both worlds—low token cost and high resolution?
Tried the math on it. Slicing works for readability, but it's not cheap. A 2540x18410 page needs 8 tiles of 2540x2576, and each one is about 8,700 tokens (w x h / 750), over the 4784 cap per image, so each tile gets shrunk again anyway.
8 tiles at 4784 is around 38,000 tokens. That's 3x more than the text version at 11,372, and still a bit blurry. For tall pages text wins on both, for one screen of UI the single image is the steal you said.
Wow, those numbers really put it into perspective. 38,000 tokens for a blurry stitched image vs. 11,372 for perfect text? That's a massive 'nope' for tall pages.
It perfectly explains why text extraction is still king for data processing, while vision is only a lifesaver for spatial UI debugging. We're basically paying a massive token tax just to see where something is on the screen.
"Context window full of noise" is the exact test I've been missing.
Most agent tests use a clean context: 5 messages, 3 tools, 1 clear goal. But production is 50 messages, 10 tools, 3 conflicting goals, and a user who pasted a screenshot of a spreadsheet instead of text.
The only way I've found to simulate that is to record real production sessions and replay them with the same noise. It's not perfect, but it catches the "agent gets confused by irrelevant information" failure mode.
Does anyone know if it’s possible to run the server locally in Qoder?
Yes, you can run an MCP server locally with Qoder. It supports the standard stdio transport, so you just configure the path to your local script (e.g., python /path/to/server.py) in the MCP settings, and Qoder will launch it as a child process.
If your server runs over HTTP (using the SSE transport), you can point Qoder to your localhost:port endpoint instead. Just make sure the server is running before you start Qoder!
Some comments may only be visible to logged-in visitors. Sign in to view all comments.