I built a RAG prototype, tested it with a handful of documents, and felt good about it. Then real users showed up. The questions got messier, the documents got longer, and the answers started drifting. One user asked about our refund policy and got instructions for resetting their password. Another asked about API rate limits and got a paragraph about billing.
That was when I realized RAG is not just about connecting a vector database to an LLM. The quality of the whole system depends on how you split content, how you retrieve it, and how you choose the final context before sending it to the model.
The model was not the problem. The pipeline around the model was.
Related: If you want the core architecture first, read my practical RAG guide. If you are building the product end to end, Building an LLM App: A Practical Guide From Prototype to Production is a useful companion.
Try it yourself: All the comparisons in this post were generated by running a RAG quality demo script against
gpt-4o-mini. It takes one long policy document, chunks it two ways, retrieves with two strategies, and reranks — then compares the answers side by side. You can install it withpip install requests rich numpy scikit-learnand run it to test your own documents.
What RAG Quality Actually Means
When I talk about RAG quality, I mean two things:
- the model answers accurately,
- the answer is grounded in the right source material.
Those are related but not identical. A polished answer that cites the wrong document is still a bad answer. A technically correct answer that ignores the most relevant chunk is also a failure. Good RAG quality means the system finds the right information and uses it well.
In practice, that depends on three layers:
- Chunking — how your source content is broken up.
- Retrieval — how the system finds candidate passages.
- Reranking — how the system chooses the best passages before generation.
When my RAG app started giving wrong answers, I traced the failures back to these three layers every single time.
Why Chunking Matters More Than I Expected
Chunking sounds boring. It had the biggest impact on my answer quality.
If chunks are too large, the relevant sentence gets buried in a wall of text. If chunks are too small, you lose the context the model needs. If chunks break in the wrong place, the system splits a definition from the example that explains it.
I had originally chunked everything at 500 characters. It worked fine for my test set of short FAQ entries. It fell apart completely on our longer product docs and policy pages.
The rule of thumb I settled on
- Use semantic boundaries when you can: headings, paragraphs, sections, FAQ entries.
- Keep chunks large enough to be useful, but not so large that retrieval becomes fuzzy.
- Add a bit of overlap when a topic flows across two chunks.
For example, our product policy document worked much better when chunked by section than when sliced every 500 characters. Our technical docs needed section-aware chunking so context stayed intact.
Chunking mistakes I made
- Splitting in the middle of a table or list.
- Using one chunk for an entire long document.
- Creating tiny chunks that read like sentence fragments.
- Ignoring headings and metadata during the split.
Chunking is not glamorous, but it was the easiest way to improve retrieval quality quickly. I fixed chunking first in almost every debugging session.
Retrieval: Finding the Right Context
Once my content was chunked well, the next challenge was retrieval. The job is simple in theory: given a question, find the most relevant chunks. In practice, this is where many of my failures lived.
What good retrieval looks like
A strong retrieval system does not just match keywords. It understands similar meaning, related concepts, and useful context. That usually means working with embeddings, but not only embeddings.
Techniques that helped me
Hybrid search. Vector search handles semantic similarity well, but keyword search is still valuable when exact terms matter. I combined both. This was especially helpful when users searched by product names, included error messages, or used exact terminology from our docs.
Metadata filters. Our documents have metadata: product version, document type, department. Using those as filters dramatically improved relevance. A question about “version 3 billing” should not return version 2 docs, and metadata filters prevented that.
Query rewriting. Sometimes the user’s question is vague or poorly phrased. A rewritten query helps the retriever understand intent. For example, “reset access” becomes “How to reset workspace access permissions in AcmeDesk.” That small change improved results more than I expected.
Better top-k selection. Fetching more chunks does not always help. Too many results add noise. Too few miss the answer. I tuned the retrieval window deliberately instead of defaulting to “top 5.”
Why Reranking Was Worth the Extra Step
A retriever gives you a shortlist. A reranker helps you choose the best items from that shortlist. I added reranking after noticing that the top semantic match was often not the best match.
For example, my retriever would return three passages:
- one that shared a lot of vocabulary with the question,
- one that actually answered the question,
- one that was broadly related but incomplete.
Without reranking, the model got the wrong context first and produced a plausible but incorrect answer. With reranking, the system reordered the candidates based on deeper relevance before sending them to the LLM.
When reranking helped most
- Long documents with many similar sections.
- Many near-duplicate chunks.
- Mixed-content knowledge bases.
- Questions that needed precision rather than broad coverage.
When reranking was overkill
- Tiny knowledge bases where there were only a few chunks to choose from.
- Very simple question answering.
- Early prototypes where I needed to ship fast.
I started without reranking and added it once retrieval issues became visible. That was a reasonable path, but I should have added it sooner. The quality jump was significant.
For a broader view of model customization choices, see Choosing Between Fine-Tuning, LoRA, and RAG.
Proving It: Real Output from Each Improvement
I ran a demo against gpt-4o-mini to prove each improvement individually. The setup: one long policy document (8 sections covering returns, shipping, products, API limits, support, privacy, and billing), chunked two ways, retrieved with two strategies, with and without reranking. Same model, same document — only the pipeline changes.
Test 1: Fixed Chunking vs Semantic Chunking
Question: “What is the return policy for enterprise bulk orders?”
The enterprise exception clause (90-day window, account manager coordination, 15% restocking fee) sits right after the general return policy in the source document.
Fixed chunking (300 chars): “Enterprise bulk orders (10+ units) have a 90-day return window and must be coordinated through the account manager.”
Semantic chunking (section-aware): “Enterprise bulk orders (10+ units) have a 90-day return window and must be coordinated through the account manager. Restocking fees of 15% apply to opened enterprise orders.”
Fixed chunking cut the chunk boundary right before the restocking fee clause. The model answered correctly but incompletely — it missed the 15% fee because it was not in the retrieved context. Semantic chunking kept the full exception paragraph together and got the complete answer.
Test 2: Vector-Only vs Hybrid Retrieval
Question: “What are the API rate limits for the Pro tier?”
Both strategies found the right chunk and produced the correct answer: “1,000 requests per minute and 50,000 per day.” But the retrieval scores told a different story.
| Strategy | Top Result | Score | Correct? |
|---|---|---|---|
| Vector-only | API Rate Limits section | 0.470 | Yes |
| Hybrid (semantic + keyword) | API Rate Limits section | 0.462 | Yes |
In this case, both worked because “API rate limits” appeared verbatim in the document. The hybrid approach matters more when users use different wording than the source — for example, searching “request throttling” instead of “rate limits.”
Test 3: Without vs With Reranking
Question: “Does CodeBuddy support Kotlin?”
This is where reranking made a visible difference. Without reranking, the top result was the “Technical Support” section — because “support” in the question matched “support” in that section semantically. The actual answer (in the “Supported languages” chunk) was ranked #3.
Without reranking — top 3 chunks:
- Technical Support section (score: 0.235) — wrong section
- CodeBuddy overview (score: 0.152)
- CodeBuddy supported languages (score: 0.123) — the right chunk, ranked last
With LLM reranking — reordered top 3:
- CodeBuddy supported languages — promoted to #1
- CodeBuddy overview
- Technical Support section — demoted to #3
The reranker understood that “Does CodeBuddy support Kotlin?” is asking about language support, not technical support, and promoted the correct chunk.
Test 4: Full Pipeline Comparison
I ran all 5 test questions through both pipelines — basic (fixed chunks + vector only) vs improved (semantic chunks + hybrid + rerank):
| Question | Basic Pipeline | Improved Pipeline |
|---|---|---|
| Enterprise return policy? | Missing restocking fee | Complete answer with 15% fee |
| API rate limits for Pro? | Correct | Correct |
| CodeBuddy pricing? | Raw list format | Clean formatted answer |
| Does CodeBuddy support Kotlin? | “Not found in context” | “Not found in context” |
| Late invoice payment? | Correct (1.5% fee) | Correct (1.5% fee) |
The Kotlin question returned “Not found in context” in both pipelines — a limitation of TF-IDF vectorization, which struggles with single rare terms. A production embedding model (like text-embedding-ada-002) handles this correctly. That is itself a useful data point: your choice of embedding model matters as much as your retrieval strategy.
Real Failures and How I Fixed Them
Internal support docs
Our support chatbot kept answering with the wrong setup steps for a feature. I traced the problem: the docs were chunked too aggressively, the retriever matched generic terms instead of the actual workflow, and the top result looked relevant but was the wrong section.
Fixing the chunk boundaries and adding a metadata filter for the specific product area improved answer quality before I changed anything about the model or prompt.
Policy and compliance search
We built a policy assistant for employees. The system retrieved the right policy page, but the answer still felt incomplete. The relevant section was split across two chunks, the exception clause was missing from the context, and the top-ranked chunk was useful but not the most authoritative one.
A reranking step plus more thoughtful chunking fixed it. The answer went from “mostly right” to “actually complete.”
The Improvement Loop That Worked
If you want to improve RAG quality without guessing, use this loop. It is what I follow now.
1. Start with a small evaluation set
Pick 20 to 50 real questions that matter to your use case. I pulled ours from actual support tickets.
2. Inspect the retrieved chunks
Do not only look at the final answer. Check whether the right context is being found. I was surprised how often the answer was wrong because retrieval was wrong, not because the model was wrong.
3. Adjust chunking first
Chunking issues are the cheapest to fix and often the most impactful.
4. Tune retrieval next
Try hybrid search, metadata filters, and query rewriting.
5. Add reranking if needed
If the right candidates are being found but not prioritized well, reranking can help.
6. Re-test the same questions
Measure the difference. Do not rely on instinct. I kept a simple spreadsheet tracking which questions got correct answers before and after each change.
That cycle is simple, but it keeps you focused on the actual failure point instead of guessing.
What Didn’t Work / Honest Limitations
Better chunking, retrieval, and reranking will not fix everything. If your source documents are wrong or incomplete, the answers will be too. If the user’s question is genuinely ambiguous, even perfect retrieval cannot resolve it. And reranking adds latency, which matters if your system needs to feel fast.
The improvement loop helps, but it requires discipline. It is tempting to skip evaluation and just tweak things until the answers “feel” better. That path leads to regressions you do not notice until a user reports them.
A Practical Summary
Improve the chunks
- Prefer semantic boundaries.
- Keep related text together.
- Add overlap where needed.
Improve retrieval
- Use hybrid search if exact terms matter.
- Add metadata filters.
- Consider query rewriting.
Improve ranking
- Add reranking when top-k results are noisy.
- Check whether the best chunk is actually first.
Improve measurement
- Build a question set.
- Review retrieved context, not just output.
- Track where the pipeline fails.
Further Reading:
Final Thoughts: Fix the Pipeline, Not the Model
The biggest lesson from fixing my RAG app was that the model was rarely the bottleneck. Almost every bad answer traced back to a pipeline problem: wrong chunks, missed context, or poor ranking. The model did fine when it got the right information.
If your RAG system is giving weak answers, resist the urge to swap models or rewrite prompts first. Look at the chunks. Look at what the retriever is returning. Fix the pipeline, then measure again. That sequence sounds modest, but it is usually the difference between a demo and something people actually trust.
Previously: I Built a Q&A Bot Over Our Company Docs: A Practical Guide to RAG. Next: I Shipped a RAG System to Real Users: Here’s What Production Actually Requires.
Leave a Reply