Complex Agent Memory Lost to a Naive RAG

The title of the new paper reads like a pointed question: Is complex agent memory necessary? The conclusion: on many conversational understanding tasks, a simple raw-retrieval RAG is stronger than a carefully designed memory system. When introducing this paper in a tweet, Akshay commented: "RAG is so back" (RAG is back).

The paper is by Zhengye Bao et al., titled "Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought." They propose Raw-RAG (RAG stands for retrieval-augmented generation: retrieve evidence first, then have the model answer): perform dense retrieval directly over raw conversation history, rerank, and then give the evidence to the LLM to answer. Put simply, skip memory processing and, when an answer is needed, go straight back to the original utterances, find the relevant passages, and have the model answer based on those passages. Across four benchmarks—LoCoMo, LongMemEval, MEME, and MemoryAgentBench—and seven settings, it took first place in six, outperforming the strongest agent memory systems by 0.13 to 5.53 points.

Why can the naive method win? The answer often lies in multiple interrelated facts: what changed, when it changed, and which old facts it affected. An isolated statement cannot support an answer. Compressing a conversation into independent memories severs these connections. Because Raw-RAG preserves the original utterances, it can provide more of the necessary evidence. The authors found that even when memory systems are given full access to the transformed store, their accuracy still cannot catch up with the raw evidence. Processing is inherently lossy.

A commenter under the tweet summarized it: the real pitfall of memory systems is that once a conversation is cut into isolated memories, all the connections are broken. Forgetting is not actually the main problem. Another comment was even more direct: raw passages preserve the exact wording, whereas memory systems that summarize first and store afterward may lose that one detail a later question depends on. This is probably why Raw-RAG wins.

This does not contradict the Observations that Akshay discussed in another article. He described Zep's approach: perform pattern recognition on a knowledge graph to find dependency chains across conversations. For example, three engineers separately report blockers; viewed in isolation, these are three problems, but the observation layer recognizes that they all stem from the same unfinished task. The key is that Observations do not delete the original facts; every observation can be traced back to specific messages and facts. Summaries only describe structure; they do not replace evidence. Algorithms decide the clustering, and the LLM only puts the structure into words.

So for agent memory, the simplest baseline is: first, don't lose things. Any complex memory system should first prove that it can beat direct retrieval over the original text. If you want to build abstractions on top of memory, at minimum ensure that the abstractions are traceable and do not replace the original text. Otherwise, you are just replacing raw evidence with a worse summary.

References:

• Paper: https://arxiv.org/abs/2610.30755
• Akshay's tweet: https://x.com/akshay_pachaar/status/2114659720602534014
• Akshay's article on Observations: http://x.com/i/article/2086395867582603264