RAG Is Back Again: Complex Agent Memory May Lose Out to Reading the Original Text Directly
The paper “Memory Tax” breaks the value of Agent memory systems into two accounts, positive and negative, and concludes that many complex Memory solutions are worse than doing RAG directly on raw history—and cost more. The problem lies in the curation step.
Agent Memory has become heavier and heavier over the past two years. Vector stores, summary layers, entity graphs, multi-level caches—a single chat assistant ends up carrying an entire information-governance pipeline.
The conclusion from the paper “Memory Tax” is a bit deflating: many specially designed Memory systems ultimately underperform direct RAG over raw history—and cost more.
The authors break Memory’s value into two accounts, positive and negative.
Memory Benefit is questions RAG gets wrong and Memory gets right. Memory Tax is questions RAG could originally have gotten right but that become wrong after Memory curation.
The problem clusters around the curation step. After history is summarized, rewritten, and structured, some details are simply lost; other details remain but become harder to retrieve. Reconstruction was supposed to make information more usable, but it ends up blocking the evidence.
The numbers on LongMemEval are not kind: Qwen3.6-27B with two-round RAG reaches 81.4%, and GPT-5.6 Luna reaches 90%, both higher than the corresponding complex Memory solutions.
The fix the authors propose is quite plain. Keep using Memory, but also retain a retrieval channel to raw history, and rerank the results from both sides together. This amounts to admitting that summaries make mistakes and leaving a path back to the original scene.
On X, a commenter under the discussion said they firmly support grep. The author replied with a smiley face. That attitude may not be complete, but it lines up with the paper’s conclusion: first use the dumbest method as a fallback, then talk about complex structures.
Another reminder worth copying down: before adding Memory to an Agent, establish the simplest raw-history RAG as a baseline. Don’t wait until the system is live to discover that the money spent on curation is greater than the loss from not remembering.
Xudong Han wrote at the end of the tweet: What is truly expensive is not failing to remember, but summarizing away the original information too early.
Paper link: https://arxiv.org/abs/2610.34073
发布时间: 2026-10-30 20:32