Production RAG: Three Retrieval Strategies and When Each One Wins

The demo takes a weekend: chunk the docs, embed them, wire up top-k similarity search, stuff the results into the prompt. Ask "what's our refund policy?" and the answer is perfect. Then, a few weeks after launch, someone asks "does the ENT-4402 plan include SSO?" — and the system retrieves three chunks about pricing tiers, none of them about ENT-4402, and the model writes a confident, well-structured paragraph anyway. Nobody files a bug, because the answer reads right.

That is the signature failure of retrieval-augmented generation, and it is worth naming precisely: retrieval fails silently, and generation launders the failure into fluent prose. A search engine that misses shows you bad results and you know it missed. A RAG system that misses shows you a paragraph.

So here is the mental model this post hangs on: a RAG system is a search engine wearing a language model as a face. The quality ceiling is set by retrieval; the face is what hides where the ceiling is. Every production decision that matters — and all three strategies below — is about raising that ceiling, or at least knowing where it is.

The Tragic Misconception

The naive design fails because of a belief that feels reasonable the whole time you hold it: embeddings understand my documents, so similarity search will find the right ones.

Embeddings are lossy semantic compression, and the loss lands exactly where production queries live:

• Exact identifiers are blurred away. In "does ENT-4402 include SSO?", the entire meaning concentrates in one rare token. Dense vectors are built to generalize across phrasings — which is precisely the wrong instinct for SKUs, error codes, function names, and legal clause numbers.
• Negation barely moves the vector. "Plans without SSO" and "plans with SSO" embed nearly on top of each other. The retriever cheerfully returns the opposite of what was asked.
• Similarity is relative, never absolute. Top-k always returns k chunks. There is no built-in signal for "nothing relevant exists" — when the corpus has no answer, the retriever hands the model the least irrelevant garbage, and the model, trained to be helpful, answers from it.
• Chunk boundaries cut answers in half. The condition lands in one chunk, the exception in the next; the retrieved half looks complete, and the model has no way to know a sentence was load-bearing.

None of these throw errors. All of them produce answers. That is what makes the naive version dangerous rather than merely weak — and why the fix is not a better prompt. It is a better retrieval strategy.

Strategy 1: Single-Shot Vector RAG

The demo architecture, promoted honestly:

What it buys you. One embedding call and one index lookup — retrieval in tens of milliseconds, the cheapest cost-per-query of the three, and infrastructure simple enough to run as a pgvector table next to your existing data. Fewest moving parts, easiest to debug, easiest to evaluate.

What it costs you. Everything in the misconception section, at full strength. You can sand the edges — a similarity floor below which you refuse to answer, overlap between chunks — but these are mitigations, not fixes. The exact-match blindness and the missing no-answer signal are structural.

When it wins. A homogeneous corpus, questions that are paraphrases of the documents ("how do I reset my password?"), latency-sensitive surfaces, and answers whose blast radius is small — an FAQ widget with visible citations, not a compliance tool. It is also the mandatory first rung even when you know you will climb: it is the baseline that tells you whether the expensive machinery below actually pays for itself.

Strategy 2: Hybrid Retrieval + Reranking

The production workhorse. Two retrievers with opposite failure modes, fused, then re-scored by a model that actually reads:

The mechanics that matter. BM25 is everything dense retrieval is not: it lives on exact rare tokens, so ENT-4402 goes from the query's weakest signal to its strongest. Reciprocal rank fusion merges the two candidate lists without tuning weights. Then the cross-encoder reranker does what neither retriever can: it reads the query and each candidate together and scores actual relevance, not neighborhood-in-embedding-space. That score is the piece the naive design was missing — an absolute relevance signal you can threshold, which finally gives the system a principled way to say "nothing in the corpus answers this."

What it costs you. Two indexes that must stay in sync with the corpus and with each other. A reranking step on the critical path — cheap per candidate, but it is a model call where there was none. And meaningfully more knobs (candidate counts, fusion, thresholds), which means you now need a retrieval eval to tune against or you are turning dials blind.

When it wins. Heterogeneous corpora full of jargon, identifiers, code, or clause numbers; user queries you do not control; products where a wrong answer is embarrassing rather than merely unhelpful. Most RAG systems that matter should end up here — it is the highest quality-per-unit-complexity of the three.

Strategy 3: Agentic RAG

Stop treating retrieval as a preprocessing step and hand it to the model as a tool:

The mechanics that matter. The model reformulates bad queries instead of failing on them, decomposes multi-hop questions ("compare the termination clauses in the Acme and Initech contracts") into sequential searches where each hop depends on the last, chooses which index to hit, inspects what came back, and — because the loop has an explicit exit — can decide the corpus does not contain the answer. The retrieval strategies above are static pipelines; this one adapts per question.

What it costs you, honestly. Latency moves from milliseconds to seconds or tens of seconds. Cost multiplies by the number of loop steps. You inherit run-to-run variance — the same question can take a different retrieval path on different runs, the same class of problem I wrote about in ensemble scoring. Evaluation gets harder: you are now judging trajectories, not just answers. And the failure modes get stranger: loops that over-search, agents that burn ten calls on a question the FAQ answered, and a larger prompt-injection surface, because retrieved content now steers a tool-wielding loop rather than a single completion.

When it wins. Genuinely multi-hop questions, corpora spread across systems (docs plus database plus tickets), low query volume with high stakes, and users who will wait for a researched answer. It is the right shape for a research assistant. It is the wrong shape for a search box.

Choosing: Match the Strategy to the Question Shape

	Vector RAG	Hybrid + rerank	Agentic
Retrieval latency	Tens of ms	Hundreds of ms	Seconds to tens of seconds
Cost per query	Lowest	Low-moderate	Multiplied by loop steps
Infrastructure	One index	Two indexes + reranker	Pipelines + tool loop + guardrails
Question shape	Paraphrases of the docs	Jargon, identifiers, messy queries	Multi-hop, cross-system
"No answer" handling	Weak (similarity floor)	Real (reranker threshold)	Explicit (agent can refuse)
Signature failure	Silent wrong answer	Stale index, tuning drift	Cost, variance, weird loops

The strategies are an escalation ladder, not a menu. Start at rung one even if you are sure you need rung three, because each rung is the eval baseline for the next — and climb only when a measured failure demands it, using the same eval-loop discipline that applies to prompts: a golden set of questions, a score, one change at a time. "Our retrieval feels weak" is not a reason to go agentic. "Recall on identifier-shaped questions is our top failure cluster" is a reason to add BM25.

What "Production-Grade" Means on Every Rung

The strategy choice gets the attention, but most RAG failures in the wild are failures of the surrounding system:

• Evaluate retrieval separately from generation. An answer-level eval cannot tell you which half failed. Keep a golden set of question-to-chunk mappings and measure whether the right chunks were retrieved at all — otherwise you will tune prompts to compensate for a broken retriever.
• Wire and test the refusal path. "I don't know" must be a designed outcome with its own eval cases, or the system will never say it.
• Show citations users can click. Not decoration — it is the only mechanism by which a wrong answer gets caught by the person reading it.
• Treat index freshness as a pipeline, not a batch job you ran once. Documents change; embeddings do not re-embed themselves. A stale index is the system confidently answering from last quarter's policy.
• Filter permissions before retrieval. In a multi-tenant corpus the index will happily return another tenant's chunks; access control belongs in the retrieval query, never in a prompt instruction asking the model to please not use them.
• Log the retrieved chunks with every answer. Debugging a bad RAG answer without knowing what the model was shown is archaeology.

Key Lessons

The ceiling is retrieval; the face hides the ceiling. Fluency is not evidence of grounding. Every improvement that matters happens before the model generates a single token.

Escalate with evidence, not fashion. Each rung of the ladder buys quality with latency, cost, and operational surface. The eval that proves you need the next rung is the same eval that proves the climb worked.

Give the system a way to say nothing. The most expensive answers a RAG system produces are the confident ones to questions its corpus cannot answer. A relevance threshold or an agent's explicit refusal is worth more than any prompt engineering.

Judge retrieval and generation separately. They fail differently, they are fixed differently, and a single end-to-end score lets each hide behind the other.

Related Reading

• Prompt Engineering Without Guesswork — the eval loop that decides when to climb the ladder
• Taming LLM Variance: Ensemble Scoring with Outlier Rejection — the run-to-run variance you inherit the moment retrieval goes agentic