RAG Can't Find Appendix G: The Problem Isn't the Vector Store

Standard RAG splits documents into chunks, embeds them, and stores them in a vector store. The query is embedded too, and the top-k most similar chunks are passed to the LLM. The pipeline is familiar, and the problem is old: similarity answers “does it look alike,” while relevance answers “can it answer the question.” The two often do not coincide.

Avi Chawla gives a concrete example in the tweet. Five annual reports from the same company will most likely share a template, with only the numbers changing each year. A question about FY2022 may land very close to a chunk from FY2025, so the LLM may pull the answer from the wrong year. Cross-references are even messier. One chunk says “see Appendix G,” but that appendix table does not look anything like it textually, so top-k will almost certainly not retrieve Appendix G.

Vectorless RAG takes another path. It removes the embedding model and vector store, replacing similarity search with LLM reasoning. In preprocessing, it parses document layout and generates a ToC tree index. Each node holds a section title, page range, and short summary. The entire tree must fit into the LLM's context.

At query time, the LLM receives both the question and the tree. It reads section titles and summaries and selects the few sections most likely to contain the answer. For a question about debt, it may select “Financial Summary” and also “Appendix G,” because Appendix G's summary says the debt details are there. The system then pulls the full sections and checks whether the evidence is sufficient. If so, it answers based on the full sections and provides page citations. If not, it goes back to the tree and continues opening nodes. Chat history is also in context. If you ask about assets and then ask about liabilities, it returns to the same report.

The video in Avi Chawla's original post draws the two pipelines side by side:

The cost is direct. Every query requires running LLM calls over the tree. Within the same run, top-k vector search is cheaper—though, of course, not guaranteed to be more accurate. The tree is only useful when documents have real structure. The applicable boundary is clear: long, structured documents, contracts, and manuals suit the tree; large volumes of short, unrelated text should continue to use a vector store. Commenter Mohammad Vazin said that for millions of short tickets, BM25 plus a reranker is more cost-effective. Avi agreed in a reply: short tickets have no section structure for the tree to use. Dusan noted that each level down the tree is another LLM call, so latency on large manuals can pile up quickly.

Year confusion also has a cheaper fix. Commenter Aakash Verma said that filtering by fiscal-year metadata before top-k usually solves the problem of FY2022 and FY2025 getting mixed up. The scenario that truly needs a tree is cross-references such as “see Appendix G.” Another practical issue: ToC headings OCR'd from scanned documents may be wrong. Avi added the limitation in a reply: the tree made of titles and summaries must fit into the LLM context, so very large documents or large collections need to be narrowed to a specific document or subtree first.

The speed account cannot be ignored either. Commenter Subhash Dasyam mentioned that recursive calls block, and without prefix caching, several minutes per query in interactive settings is hard to accept. He cited paper results saying GPT-5-mini scored 34 points higher than base GPT-5 on OOLONG, but that benchmark leans toward retrieval; on math tasks, overhead eats the gains. Another commenter, joel, used SQLite keyword search plus agent memory and, on LongMemEval, a public 500-question test, got the correct conversation in the top five 95.8% of the time. Gafae Boran pointed out that maintaining an index with Markdown tables and opening documents on demand is essentially the same pattern: keep the index small and fetch details on demand.

The most memorable judgment from this discussion is not complicated. Document structure itself is data. Throwing every question at vector similarity means giving up signals such as sections, page numbers, and cross-references. But the tree is not a free lunch either. Selection comes down to two things: whether the documents have structure, and whether the query is willing to pay for multiple LLM calls. A more robust engineering path may be to start with cheap retrieval and climb the tree only when the evidence is insufficient.

References: Avi Chawla's original post https://x.com/_avichawla/status/2116069319205663159 ; breakdown of MIT Recursive Language Models http://x.com/i/article/2029797316404899840 .