I've been studying and building RAG systems for a while, and I've noticed a pattern.

Most tutorials stop once the demo works.
But production introduces a completely different set of problems.
In my experience, the first issues usually aren't model-related.
They're system-related.

Examples:

• Retrieval returns technically relevant but practically useless context
• Costs grow much faster than expected
• Evaluation becomes difficult and inconsistent
• Model updates introduce subtle behavior changes
• Latency becomes a real constraint

The more I worked on RAG systems, the more I realized that many failures happen between components rather than inside the model itself.

That shifted how I think about AI engineering.

Instead of asking:
"Can this work?"

I started asking:
"What happens when it fails?"

I've been organizing everything I've learned into an open-source project called AI Model Atlas, focused on AI system design rather than specific frameworks.

GitHub:

For people who have deployed RAG systems:
What was the first production issue that surprised you?