AI ENGINEERING / RAG
Retrieval Evaluation Before Model Tuning
How to measure retrieval quality before reaching for a bigger model.
Key Insight: When a RAG system gives a bad answer, the instinctive fix is usually "try a better model" or "tune the prompt." In my experience, the actual cause is retrieval more often than either of those — and no amount of model improvement fixes an answer built from the wrong context.
A language model can only answer as well as the context it's given. If retrieval surfaces the wrong chunks, or the right chunks buried under noise, even the best available model will produce a fluent, confident, wrong answer — because from the model's perspective, it did its job correctly given what it was handed. Diagnosing that as a model problem and reaching for a bigger model wastes effort and money without fixing anything.
Measure retrieval directly, separately from the final answer
The discipline that catches this early: evaluate retrieval quality on its own, before it's obscured by the model's ability to write a fluent-sounding answer regardless of whether the underlying context was right. Build a test set of realistic questions, each with the specific chunks or documents that should have been retrieved to answer them correctly — this is usually assembled from real user questions plus domain expert review, not invented in the abstract.
Against that set, measure precision (of what was retrieved, how much was actually relevant) and recall (of what was relevant, how much was actually retrieved). A system with strong recall but weak precision is burying the model in noise; a system with weak recall is missing the answer entirely, and no prompt engineering downstream can recover information that was never retrieved.
The failure modes retrieval evaluation actually surfaces
Running this consistently surfaces specific, fixable problems: chunks that are too large, so the relevant sentence is diluted by irrelevant surrounding text and scores lower on similarity than it should; chunks that are too small, losing the context that made them meaningful in the first place; embedding models that don't handle domain-specific terminology well, meaning a policy number or product code doesn't match semantically the way it should; a missing keyword-search component, so queries with exact terms (error codes, specific names) underperform because pure vector similarity isn't the right tool for that kind of lookup.
Every one of these is a retrieval-layer fix — better chunking strategy, hybrid search, a domain-adapted embedding model — not a reason to reach for a larger, more expensive generation model.
Where reranking earns its cost
Once basic retrieval is solid, reranking the top-k candidates with a cross-encoder before they reach the generation model is one of the highest-value, lowest-risk improvements available, because it directly addresses the gap between "semantically similar" and "actually answers this question" — two things that correlate but aren't identical. Measuring the effect requires the same evaluation set: track precision at the top few results before and after reranking, since that's what the model actually sees.
Only tune the model once retrieval is solid
Model-level improvements — better prompting, a larger model, fine-tuning — absolutely have their place, but they belong after retrieval evaluation shows the model is actually being given good context and still producing weak answers. Applied before that, they're solving the wrong layer of the problem, and it's easy to convince yourself a bigger model "helped" when what actually happened is a more capable model partially compensating for bad context — a fragile improvement that degrades again as soon as the document set changes.
Bad answer → is retrieval precision/recall low? yes → fix chunking / hybrid search / reranking no → then investigate prompting or model choiceKey takeaways
Evaluate retrieval quality directly and separately from final answer quality — don't let a fluent wrong answer disguise a retrieval problem as a model problem. Build a real test set with known-correct source documents, not just a handful of manual spot checks. Use precision and recall to diagnose specific, fixable retrieval issues: chunking size, missing hybrid search, weak embeddings for domain terms. Add reranking once basic retrieval works — it's high value, low risk. Only invest in model-level tuning after retrieval evaluation shows the context being provided is actually good.