The 6:14 AM Wake-Up Call That Changed Everything
Picture this: It's 6:14 AM, your Slack is exploding, and your shiny new RAG + MCP agent just told an on-call engineer that a critical pipeline failure is "covered under a Tier-2 SLA with a 4-hour response window." The real SLA? 30 minutes. The job had been dead for 90.
That's exactly what happened to developer Sudip P. when he shipped his RAG + MCP agent to production. What worked flawlessly in demos with two test queries became a liability with 80 real users pasting actual Slack threads.
If you're building AI agents or working with RAG systems, this cautionary tale offers invaluable lessons about the gap between demo magic and production reality.
The Architecture That Should Have Worked
The concept was solid: a small LLM router classifies user queries as either Knowledge requests (handled by RAG with hybrid search and cross-encoder reranking) or Action requests (handled by MCP tools with validation and timeout handling). Both paths converge at a frontier model for synthesis, then pass through evaluation and logging before returning a response.
On paper, and in demos, it was elegant. In production? That's where things got interesting.
Breakage #1: When Vector Search Gets It Wrong
What failed: The model didn't hallucinate – the retriever handed it the wrong information. "Tier-1 SLA for streaming" and "Tier-2 SLA for batch" are nearly identical in vector space, but completely different to a human at 3 AM dealing with an outage.
The lesson: Vector similarity isn't the same as relevance. Embeddings smooth over the exact words that actually matter most.
The fix: Implement hybrid search combining BM25 (keyword matching) with vector search, then rerank the top candidates with a cross-encoder model. BM25 catches literal terms like "Tier-1" and "streaming" that embeddings blur together, while cross-encoders evaluate query-chunk pairs together to catch "semantically close but factually wrong" cases.
def retrieve(query, k=5):
candidates = hybrid_search(query, k=20) # BM25 + vector
reranked = rerank_model.rerank(query, candidates, top_n=k)
return reranked
Breakage #2: The Silent Null That Spelled Disaster
What failed: An MCP tool called a job status API that timed out and returned null. The model interpreted null as "no issues" and confidently told the user everything was fine. The job had actually been dead for 90 minutes.
The lesson: AI models need explicit structure to distinguish between "no data" and "everything is OK." Raw nulls are dangerous.
The fix: Wrap every tool call in structured results with explicit validation, timeouts, and ok/error flags:
def get_job_status(job_id):
try:
validate(job_id)
data = call_api(job_id, timeout=5.0)
return {"ok": True, "data": data}
except TimeoutError:
return {"ok": False, "error": "upstream_timeout"}
except Exception as e:
return {"ok": False, "error": "upstream_failure", "detail": str(e)}
When a tool returns an explicit error structure, the model can provide helpful responses like "I couldn't reach the job system – please retry" instead of false confidence.
Breakage #3: When Simple Routing Meets Real Users
What failed: The demo router was 12 lines of if statements checking for keywords like "job" or "sla." Real users don't type clean queries – they paste messy Slack threads and ask complex, contextual questions.
The fix: Replace keyword matching with a small LLM as a router. It can handle the ambiguity and context that keyword matching simply can't.
The Critical Success Factors
Beyond fixing these specific breakages, several system-wide practices emerged as essential:
- Build evaluation from day zero: Without evals, you have no clue what's actually working
- Cache everything you can: Reduce latency and API costs
- Log chunk IDs and tool calls: When users complain, you need instant visibility into whether retrieval or synthesis failed
- Monitor tool call success rates: The first tool to drift is never the one you expect
The Takeaway for Prompt Engineers
This story illustrates a fundamental truth about AI systems: demos lie, and production finds your assumptions. Whether you're crafting prompts, building RAG systems, or designing agent architectures, the gap between controlled testing and real-world usage is where most projects succeed or fail.
The key isn't perfection in demos – it's building systems robust enough to handle the messy, unpredictable nature of real user interactions. That means structured error handling, hybrid approaches that cover edge cases, and comprehensive monitoring from day one.
As Sudip learned the hard way: "Every box [in the architecture] exists because something broke without it."
Source: Based on the article "5 Things Broke When I Shipped a RAG + MCP Agent to Production" by Sudip P., originally published on Towards AI.