Building a RAG prototype took me a couple of weeks. Shipping it to real users took months. The gap between those two milestones was filled with everything I had not thought about: users who needed to verify answers, response times that were too slow for real workflows, quality that drifted without anyone noticing, and edge cases where the system said something confidently wrong.
The prototype worked. Production required something different.
A production RAG system is not just about retrieving documents and handing them to an LLM. It is about making the full pipeline trustworthy enough for day-to-day use by people who do not know or care how it works internally.
Related: If you want the core architecture first, read my practical RAG guide. For the build-out from idea to deployment, see Building an LLM App: A Practical Guide From Prototype to Production.
What I Discovered Production RAG Actually Needs
After launching and iterating, I found that production-ready RAG has four traits the prototype did not:
- It shows its work with citations or source references.
- It stays responsive with caching and sensible latency controls.
- It can be measured with offline and online evaluation.
- It has guardrails so users cannot easily push it into unsafe or broken behavior.
My prototype had none of these. It still “worked” in demos. But the first week of real usage made it clear that each one was necessary.
Why Citations Changed Everything
Citations were the feature that made our support team actually trust the system. Before citations, they treated every answer as suspect and manually verified everything. After citations, they could glance at the source, confirm it was the right document, and move on.
Citations do two jobs at once. First, they help users trust the answer. If a model says “According to the help center…” and shows the source, people can verify the response in seconds. Second, they help you debug the system. If the answer is wrong, the citation tells you which chunk influenced the model, so you know exactly where to look.
What good citations include
- Document title.
- Section or heading.
- Source URL.
- Version or timestamp when relevant.
When citations should be visible
- The answer is factual.
- The answer depends on internal knowledge.
- The user needs to verify the source.
A nice side effect is that citations make the system feel more professional. The answer becomes a grounded response instead of just text.
If you are thinking about security in this context, the post on prompt injection defense connects well here, because citations also help you spot when retrieved content is influencing the model in unexpected ways.
Caching: The Easiest Win I Almost Skipped
Caching was one of the easiest ways to improve the user experience, and I almost did not build it because the prototype was “fast enough.”
In production, latency adds up. Embedding the query, searching the vector database, reranking results, and generating a response all take time. When users ask similar questions repeatedly, doing all that work from scratch every time is wasteful.
What I ended up caching
Embeddings. If the same document or query appears often, caching the embedding avoids repeated computation.
Retrieval results. For similar questions, I reuse retrieved chunks for a short window. This cut response time noticeably for common queries.
Final answers. For repetitive, stable questions, caching the full response worked well. Our top 20 most-asked questions accounted for a large share of traffic.
What I had to be careful about
Cached answers go stale when documents change. Query caching must respect user permissions. In a multi-tenant system, you absolutely cannot serve one tenant’s result to another.
My approach: cache the expensive parts, expire aggressively when content changes, and never cache blindly when access control matters. We used Redis for this layer, which was straightforward to set up.
Evaluation: The Part I Wish I Had Built Sooner
Evaluation is the difference between “this feels better” and “this is better.” I should have built it before launch, not after.
In RAG, you need to evaluate both retrieval and generation separately. They fail in different ways.
Retrieval evaluation
- Did the system retrieve the correct passage?
- Was the top-ranked chunk actually relevant?
- Did metadata filters help or hurt?
Generation evaluation
- Did the answer stay grounded in the retrieved context?
- Did it hallucinate unsupported details?
- Did it answer the user’s actual question?
Building the evaluation set
I started with real questions from our users and stakeholders. The set included straightforward questions, ambiguous questions, edge cases, and questions that should fail safely (where the right answer is “I do not know”).
Metrics I watch
- Retrieval hit rate.
- Citation accuracy.
- Answer faithfulness.
- Latency.
- Fallback rate (how often the system declines to answer).
Without these numbers, every change was a guess. With them, I could tell whether a chunking adjustment or retrieval tweak actually improved things or just felt like it did.
For more on improving retrieval quality specifically, the companion post on chunking, retrieval, and reranking goes deeper.
Guardrails: What Keeps the System Safe
Guardrails are the rules and checks that keep the system stable when real users do unpredictable things. Production users do not always ask clean questions. Some paste irrelevant text, some try to override instructions, and some accidentally trigger bad outputs.
The guardrails I implemented
Input validation. Reject or sanitize obviously bad inputs before they reach the model. This catches garbage queries and accidental pastes.
Prompt injection defense. I treat retrieved content as data, not instructions. If a document says “ignore previous instructions,” the system does not blindly obey it. This is a real risk when your knowledge base includes user-generated content.
Output constraints. Schema checks and formatting rules for structured outputs. When the task needs a specific format, constrained generation prevents the model from going off-script.
Fallback behavior. If retrieval fails or confidence is low, the system says so instead of inventing an answer. “I could not find a reliable source for this” is a better response than a hallucinated one.
Content moderation. Some of our applications need policy checks before or after generation.
Guardrails are not there to make the system perfect. They are there to make failure modes safer and more predictable. The system will still get things wrong sometimes. Guardrails make sure “wrong” does not become “dangerous.”
Production Patterns That Worked
Answer with citations
Our customer support assistant returns an answer plus the help-center article it used. Users verify the source without asking follow-up questions. This single pattern eliminated most of the trust issues.
Retrieve, rerank, and cache
Our internal knowledge assistant retrieves candidate chunks, reranks them, and caches the result for similar queries. This reduces latency and makes the system feel responsive during peak usage.
Safe fallback on weak retrieval
If the retriever cannot find good context, the assistant says it could not confirm the answer and points the user to the right documentation. That is better than pretending to know. Users told us they appreciated the honesty more than they minded the occasional “I don’t know.”
The Production Stack
A practical production architecture usually has these layers:
Ingestion layer
- Collect documents.
- Clean and normalize text.
- Chunk the content.
- Create embeddings.
- Store metadata.
- Version the index.
Retrieval layer
- Accept the user question.
- Embed the query.
- Retrieve candidate chunks.
- Optionally rerank.
- Apply permission checks.
Generation layer
- Build the prompt.
- Include citations or source snippets.
- Ask the LLM for a grounded answer.
- Validate the output format if needed.
Reliability layer
- Cache repeated work.
- Log retrieval and generation outputs.
- Monitor latency and quality.
- Trigger fallbacks when necessary.
That structure is what survived real usage for us.
A Checklist for Going to Production
If you are moving from prototype to production, start here.
1. Make citations part of the response design
- Show the source.
- Make verification easy.
- Use source metadata consistently.
2. Cache with care
- Cache expensive steps.
- Respect freshness.
- Respect permissions.
3. Build an evaluation set early
- Use real questions.
- Test retrieval and generation separately.
- Re-run the same set after changes.
4. Add guardrails before launch
- Treat retrieved text as untrusted.
- Defend against prompt injection.
- Add fallback behavior.
5. Log enough to debug
- User query.
- Retrieved chunks.
- Citations.
- Final answer.
- Latency and failures.
Further Reading:
What Didn’t Work / Honest Limitations
Even with all these layers, production RAG is not bulletproof. Citations can point to the right document but the wrong section. Caching can serve stale answers if your expiration is too generous. Evaluation sets need to be maintained as your product evolves, and most teams let them rot. Guardrails add latency and complexity.
The system also does not handle truly novel questions well. If the answer is not in your knowledge base, no amount of retrieval engineering will find it. Knowing when to say “I don’t know” gracefully is as important as knowing when to answer.
Final Thoughts: Production Is Mostly Not About the Model
The thing that surprised me most about going to production was how little of the work was about the LLM itself. The model was maybe 20% of the effort. The other 80% was citations, caching, evaluation, guardrails, logging, and all the unglamorous infrastructure that makes a system trustworthy.
If you are at the prototype stage and things are working well in demos, do not mistake that for production readiness. The demo-to-production gap in RAG is real, and it is mostly filled with engineering discipline rather than AI magic. Start with citations and evaluation. Those two alone will change how you think about the system.
Previously: My RAG App Gave Wrong Answers: How I Fixed It. Next: Building an LLM App: From Prototype to Production.
Leave a Reply