Memory Tax: Before Adding Memory to AI, Run Naive RAG First
A new paper finds that in many cases, specialized memory systems cannot beat naive RAG that directly pulls up raw conversation logs, and they cost more. It separates memory gains and losses into two terms, explaining an often-overlooked problem: building memory itself loses information.
RAG wins again.
Researchers from the University of Illinois, Tsinghua University, Renmin University of China, Verdent.ai, and the University of Waterloo have released a paper titled 'Memory Tax: Dissecting the Net Value of Agent Memory.' They conducted a large-scale evaluation on three benchmarks—LongMemEval, PersonaMem, and MINT-Eval—using multiple base models, and found an awkward fact: in many cases, specialized memory systems perform worse than naive RAG that directly retrieves raw conversation logs, while costing more.
The paper breaks a memory system's value into two parts. First, memory benefit: questions the memory system answers correctly but RAG gets wrong. Second, memory tax: questions RAG answers correctly but the memory system gets wrong.
Where does the memory tax come from? Because building memory is itself an act of rewriting. When extracting summaries, some facts are dropped or distorted; others survive but become harder to retrieve. One commenter gave an example: summarizing 'moved from Austin to Denver in March' as 'lives in Denver' loses the date. In addition, after changing the base model, memory system performance fluctuates more.
There is a simple fix: retrieve from both the raw history and the memory, then rerank the results together; this can significantly reduce the memory tax. The authors also offer a more pragmatic recommendation: before adding a memory layer to an agent, first compare naive retrieval over raw history with a memory system on your own data.
One commenter noted that this conclusion needs context: the paper's actual conclusion is more cautious than social media retellings. Some commonly used memory benchmarks are themselves highly favorable to retrieval. A simple keyword (IDF) baseline that does not use embedding vectors can exceed 50% accuracy on some benchmarks, and hybrid RAG can reach around 70%. That leaves little room for memory systems to add benefit, while their losses become visible. On web tasks, however, distilling long trajectories into workflows can still outperform naive RAG.
'Many memory layers are just lossy summarization plus more steps,' one commenter summarized. Another observation: the raw conversation log is the only version that no one has summarized incorrectly; memory systems all perform lossy compression with a prediction of 'what you will ask later,' while RAG does not make that bet.
There is also a practical consideration: a memory layer should be reconstructable from the raw conversation. Otherwise, switching to a different extraction model becomes a data migration, and discarded facts are already gone before the new model has a chance to use them.
The value of this paper lies in introducing the perspective of the 'memory tax.' While everyone is discussing how to make memory more complex, it offers a simple, cheap, interpretable baseline: directly retrieve the raw history. Get that working first, then decide whether to add a memory layer.
发布时间: 2026-10-28 09:20