Agent Memory May Have Been Overcomplicated: No Memory Extraction, Yet Takes First in 6 Test Groups
A paper and the open-source project Obelisk point in the same direction: don't rush to have models summarize memory; retain the raw conversation history and retrieve it when needed. Across four benchmarks and 7 settings, RAW-RAG takes first in 6 settings, leading by as much as 5.53 points.
Agent Memory may have been overcomplicated.
A recent paper, 《Is Complex Agentic Memory Always Needed?》, ran a very direct experiment: no Memory Extraction, no complex knowledge graph, and no letting the LLM repeatedly summarize and update memory at write time. It only keeps the raw conversation history, retrieves it when needed, and lets the LLM reason over it on its own.
This approach is called RAW-RAG. It took first place in 6 out of 7 test settings across four benchmarks, including LoCoMo and LongMemEval, leading the strongest Agentic Memory system it was compared against by as much as 5.53 points. A benchmark can be understood as a public test set, and the 7 settings are tests under different tasks and conditions. This result at least shows one thing: first distilling memory into summaries and then having the Agent use them is not necessarily the better solution.
Project author Yuu mentioned in a tweet that this is also the approach they insisted on when developing Obelisk. An Agent should not only see memories that someone else has summarized for it. The reason behind a code change, a failed attempt, or the dependencies among several decisions can all be lost during summarization and information extraction. Rather than guessing what information will be worth remembering in the future, it is better to retain the complete history and let the Agent find it itself when it truly needs it.
Obelisk is an open-source Coding Agent Session retrieval system. It can directly query the historical Sessions of Agents such as Claude Code, Codex, Hermes, Pi, and Opencode, including conversations, tool calls, Subagents, and workflows. It does not rely on Embeddings and does not need to extract Memory from history in advance. Under the hood, it uses SQLite FTS5 and active querying by the Agent, enabling the Agent to rediscover raw evidence from the past. FTS5 can be understood as SQLite's built-in full-text search capability; it does not rely on vector retrieval.
The questions it aims to answer are very specific:
- Why was this piece of code originally designed this way?
- Where did the previous Agent leave off in its task?
- What approaches did we try before, and why did they fail?
- How did this project's architecture evolve step by step?
The project author prefers to regard Memory as a history that can be traced at any time; a summary that is constantly being rewritten can hardly play this role. For ordinary users, this is somewhat like keeping the complete chat history and operation logs rather than only a meeting summary. A meeting summary is quick to read, but it has already decided for you what is important.
In the comments, some users said that AI having reached this level really shows that the greatest truths are the simplest—the same goes for harness and memory. Others joked that git is the best memory system. This statement may not directly apply to all Agent Memory scenarios, but it grasps the same point: the raw record itself is very valuable.
The skepticism is also very grounded. Some users pointed out that for personal projects it does not matter how you do it, but enterprise projects have too much memory content; using native memory directly cannot handle it, and a single retrieval can pull out tens of MB of relevant memories, so trade-offs need to be made based on one's own scenario. The author replied that the agent can limit the retrieval scope itself. Another common concern is that if a plan was overturned midway, retrieval will bring up both old and new decisions. The author replied that the records have timestamps, and agentic search will not make mistakes on this kind of problem.
Some users also called summaries lossy compression—deciding what to discard before knowing what will be needed in the future. Retaining raw history shifts the difficulty from summarizing in advance to retrieval quality. This judgment is crucial. RAW-RAG winning 6 test settings does not mean retrieval itself has no cost. Some asked about speed; others mentioned that limited context is the real ceiling. Some users also brought up optmem; the author distinguished it by saying that optmem is actively written by the agent, whereas Obelisk queries raw session logs—the two are not the same thing.
The paper's title asks: Is complex Agentic Memory always necessary? At least judging from this set of experiments and Obelisk's implementation, the answer deserves a question mark. First retain the raw history, then let the Agent look things up itself—this may be closer to the original purpose of memory than repeatedly rewriting a summary.
Obelisk is fully open source: https://github.com/tommy0103/obelisk
发布时间: 2026-11-01 12:55