RAG Wrong Answers? Don't Rush to Change Models: Fix Retrieval First and Cut Failure Rate by 67%

When enterprise RAG gives wrong answers, the first reaction is usually to swap models or tweak prompts. A recent LangChain community spotlight featured an enterprise RAG guide from Soba Labs, where author Kacper Klusek gives a different order: fix retrieval first, then touch the model. In a set of evaluations published by Anthropic, contextual embeddings reduced the top-20 retrieval failure rate from 5.7% to 3.7%; adding contextual BM25 brought it to 2.9%; adding a reranker brought it to 1.9%. Together, those three stages cut the failure rate by 67%. These failures happen before the generative model ever sees a token.

The original discussion is in LangChain's X post, and the full guide is on the Soba Labs blog. The article breaks enterprise RAG into a sequence of clear decisions: architecture, chunking, hybrid retrieval, reranking, and evaluation. Each step has concrete numbers, rather than being a pile of configuration checklists.

First, architecture. LangChain's documentation divides RAG into three types: 2-Step RAG, Agentic RAG, and hybrid. In 2-Step, retrieval always happens before generation, latency is predictable, and the number of LLM calls is bounded; Agentic RAG lets the model decide during reasoning when and how to retrieve, which is flexible, but more loops mean more failure modes; hybrid sits in between, keeping a validation step so it can re-query when recalled content is too thin and verify answers against sources. The key choice is who controls the loop. Start FAQ and documentation bots with 2-Step. Consider Agentic for multi-source, multi-step questions. Every extra degree of freedom you give the model adds another failure mode you need to evaluate.

Chunking looks like preprocessing, but it is actually deciding the retrieval unit. LangChain's splitter documentation gives a very direct default recommendation: in most scenarios, start with RecursiveCharacterTextSplitter. It tries to preserve paragraphs, and if a paragraph exceeds the chunk size, it falls back to sentences and words. The default parameters in the knowledge base tutorial are chunk_size=1000, chunk_overlap=200, add_start_index=True. In the tutorial, a 107-page PDF was split into 516 chunks. add_start_index preserves character offsets, which you will use the first time a user asks where an answer came from.

More troublesome is that a chunk loses references when it leaves the original text. Anthropic's contextual retrieval approach adds a short context blurb to each chunk before embedding. With contextual embeddings alone, the top-20 failure rate dropped from 5.7% to 3.7%, at a one-time cost of $1.02 per million document tokens. The article also flags an easily overlooked boundary: if the knowledge base is under 200,000 tokens—roughly 500 pages of material—you can put the entire knowledge base directly into the prompt and may not need RAG.

Hybrid retrieval addresses queries that embeddings are not good at. When users search for error codes, part numbers, or clause numbers, vector retrieval may match semantically similar content whose strings are wrong. Putting a dense index and BM25 side by side and then fusing the results is an old technique that remains on the job. Weaviate's hybrid search docs state it clearly: alpha ranges from 0 to 1, where 0 is pure keyword and 1 is pure vector, and Relative Score Fusion has been the default fusion method since v1.24. In Anthropic's evaluation, contextual embeddings plus contextual BM25 pulled the failure rate from 3.7% to 2.9%, a 49% reduction relative to the 5.7% baseline. BM25 is more valuable in corpora dense with identifiers and terminology; metadata filters such as date, product, and tenant usually reduce the candidate set more cheaply than changing indexes.

Reranking is the cheapest remaining slice of accuracy. The first retrieval layer aims for recall over a large corpus; the reranker is a second model that reorders a small candidate set by query relevance. It reads both the query and the document, so it can catch relationships that the bi-encoder structure cannot. In Cohere's Rerank 3.5 announcement, it outperformed hybrid search by 23.4% and BM25 by 30.8% on a financial services dataset. Anthropic's pipeline first takes the top 150, then keeps the top 20 after reranking, bringing the failure rate down to 1.9%.

At this point, the gains from the retrieval pipeline are clear. But enterprise RAG has a second half: documents change. Products are renamed, policies are updated, yesterday's precedent conflicts with today's rules, and the system will not stay unchanged just because the code was not deployed. LangSmith's RAG evaluation tutorial breaks quality into four comparisons: correctness, whether the answer matches the reference; relevance, whether the answer matches the input; groundedness, whether the answer is grounded in the retrieved documents; and retrieval relevance, whether the retrieved documents match the input. Once separated, problems are easier to locate. An answer that is relevant but unsupported means the model is freelancing; an answer that is grounded but irrelevant means you should check retrieval. Online evaluation uses LLM-as-a-judge on real traces, and you can set a sampling rate to control cost—for example, 0.1 evaluates only one tenth of traces.

The pre-launch checks the article gives are also concrete. First, write-read round trip: after a human correction is written back, run again with a similar input and confirm that the new precedent actually appears in the recall set. Second, rerun golden-set regression when data changes, not only when code is deployed. Third, record the embedding model version for every piece of data from day one. Models get deprecated, similarity across versions is meaningless, and re-embedding a few thousand records is a migration. It is better to plan ahead than to discover it after launch.

One commenter in the discussion said fixing retrieval first is also easier to debug: you can look directly at the retrieved chunks and know why the answer is wrong. Another asked how to handle outdated or conflicting information when enterprise documents keep changing. The article does not offer a universal switch, but it does give actionable directions: online evaluation should watch groundedness and retrieval relevance, the write path should do round-trip tests, and data changes should trigger regression. Someone joked that 'fix retrieval first' is like telling RAG engineers to go clean their room first. Crude, but the order is right.

The full guide and code snippets are in the original Soba Labs article. If you only remember one thing: models will keep getting stronger, but the retrieval stack will not automatically get better. The gap in enterprise RAG is often already set before the model sees its first token.