Retriever-Free RAG Hits 65.4%, Then Gets Stuck on Who Writes the Table of Contents

The bottleneck in standard RAG is not generation; it's the retrieval step. Documents are chunked, converted into embeddings, and stored in a vector index; when a query arrives, it is also converted into an embedding, top-k results are retrieved, a reranker reorders them, and finally the best passages are stuffed into the LLM. The problem is that similarity search decides in advance what the LLM can see. If the correct passage does not make it into the top-k, even a stronger reranker later cannot recover it.

A new paper from Cornell and ASAPP takes a different approach: keep the documents and the LLM, but replace similarity search with navigation. Documents are organized into a tree, like a table of contents. A decision model walks down from the top level, asking yes/no questions layer by layer. This model does not generate text; it reads the same context and returns a probability for each yes/no question, with one forward pass per stage.

The process is roughly: first look at the top-level documents; the model puts the query together with each candidate document and asks 'Relevant?', keeping the best few; then open those documents and ask the same question of the passages inside; optionally, use a multiple-choice re-ranking step to reorder the top 20 passages. Discarded documents are not opened, and their passages are not scored.

The paper tests on Natural Questions with a corpus of 2.68 million Wikipedia passages. Its top-1 result contains the answer 65.4% of the time. Embedding search plus an 8B reranker reaches 48.0%, and GPT-5 reranking reaches 60.8%.

There is a counterintuitive detail here: the gain comes from ranking, not coverage. The top-100 from embedding search contains the answer 98.4% of the time, while the shortlist after navigation contains it 94.0% of the time. In other words, vector retrieval is better at keeping the answer in the pool, but navigation is better at putting the answer first.

The paper itself lists the limitations. It requires a reasonable hierarchy; documents pruned early are gone for good; it only skips embedding search at query time because the Wikipedia tree was built offline using embeddings. Median retrieval time is under 2 seconds, but each method was timed in different environments, so this cannot be treated as a controlled speed comparison.

Several questions in the comments are closer to real engineering than the abstract. Someone asks: Wikipedia has an existing structure, but a shared drive does not—so who writes the table of contents? Someone says that taking the wrong path in the tree and missing in the top-k are the same kind of failure, just moved to a different place: if you take a wrong branch near the root, the correct passage disappears just the same. Another person asks what to do when the corpus changes every day: the tree is built offline, new documents get assigned to the wrong branch, and until the next rebuild they may be pruned at the top level on every query. A more robust approach is also suggested: embeddings handle recall, and the decision model handles ranking, so at least things are not thrown away before the model has seen them.

An independent benchmark of Jev as a reranker is also worth looking at. On three BEIR datasets—SciFact, NFCorpus, and FiQA—with BM25 taking top-30, Jev batch has an average nDCG@10 of 0.501, Voyage rerank-3 is 0.504, Cohere rerank-v3.5 is 0.486, and Mixedbread v3.1-listwise is 0.511. On price, Jev is about $0.54 per thousand queries, Voyage about $0.50, and Cohere about $2.00. Median latency is 223 ms and p95 is close to 1.4 seconds, so its tail is worse than dedicated rerankers. What it adds is a calibrated probability that can be used directly for pruning: on SciFact, 76% of documents scored above 0.9 by Jev were judged relevant, while only 0.5% of documents that scored below 0.1 were judged relevant.

If you are building RAG, the more practical attitude is not to choose one of the two. First benchmark retriever-free RAG as a reranker and compare it with your existing vector-retrieval-plus-reranker pipeline. Whether the tree can be built, how costly a single pruning mistake is, and how often the corpus updates—these matter more than the 65.4% figure in determining whether it can go live. Paper link: https://arxiv.org/abs/2610.35868 . Jev reranker test: https://hevmind.com/writing/jev-as-a-reranker/ .