4B Model Matches GPT-5.4: The Hard Part of Agent Memory Is Judging What to Write

Giving an agent a MEMORY.md so it takes notes while working is a common practice. The problem is that many agents record a pile of logs, while key decisions are left out, so they hit the same pitfall again in the next round.

CMU's RIME work breaks this down: the memory system itself is not necessarily the bottleneck; the model's judgment about when to write and what to write is more critical. elvis recommended this work in a tweet; his notes mention that this has little to do with the memory system and more to do with the model's awareness and judgment of when to act. That judgment is worth noting.

RIME trains one model to play two roles. The actor solves tasks using existing memory, and the writer reviews after the task ends and updates the notes for the next task. The two roles share parameters. During training, the writer's reward is the average performance on the next K tasks minus the average performance on the past K tasks. It rewards the improvement brought by memory, and has little to do with how many words are remembered. In addition to task completion, the actor's reward includes an exploration reward. An LLM judge compares memory before and after the task to see whether there are new findings, corrected mistakes, specific clues, or avoidance of repetition. Rewriting synonymous sentences gets no points.

There is a counterintuitive point in this design. Training only the actor with multi-round RL using task rewards can improve problem-solving ability, but it will not improve memory quality. Training only the writer is also not enough. The paper's experiments say that the writer can retain information from the actor's trajectories, and the actor depends on the memory left by the writer. If the actor only focuses on the current task, there is nothing worth recording in its trajectory, and no matter how strong the writer is, it is useless. Therefore, the actor needs to be rewarded for exploration, for producing reusable information. The exploration reward looks at whether the memory update has actual content, and has little to do with state novelty. This idea is more general than a traditional novelty bonus and does not require hand-designing state representations.

The experiments cover two domains: Symbolic Alchemy and Codebase Adaptation. The latter comes from Continual Learning Bench; the test stream is 19 bug-fixing instances from the tablib and tenacity repositories, unseen during training. There are two metrics: cumulative reward looks at task performance, and cumulative gain looks at how much better the model is with memory than solving each task from scratch. This distinction is important. A model may have high cumulative reward but low gain, indicating that it was already strong and did not learn from history.

Results: Qwen3.5-4B achieves 10.55 cumulative reward and +2.27 cumulative gain on Codebase Adaptation. The original text says this cumulative reward is close to GPT-5.4 and Claude Opus 4.7 with in-context learning, and the cumulative gain is close to GPT-6 Astra. In the table, GPT-5.4 is 10.14 and +0.69, Claude Opus 4.7 is 10.36 and +1.49, and GPT-6 Astra is 17.25 and +2.26. Note that the rankings for cumulative reward and gain are inconsistent: Claude Opus 5.5 has the highest cumulative reward at 17.40, but its gain is only +0.90. This phenomenon shows that strong models do not necessarily learn from experience. Putting frontier models into Claude Code or Codex does not automatically solve the problem either; Codex is even worse than its underlying model GPT-5.4 under pure in-context learning. This result is not friendly to the assumption that adding a harness enables continual learning. The authors also mention that there is no distillation or SFT here, only RL with the correct rewards and curriculum setup.

The most noteworthy ablation is the exploration reward. Remove the exploration reward, and even if the writer has been trained, the cumulative gain on Codebase Adaptation becomes negative; the model with memory is worse than solving from scratch each time. Training the writer plus an exploring actor is the best of the four combinations. Figure 10 shows that during joint training the exploration score is roughly stable, because after the writer becomes stronger it is harder for the actor to find new information, and the exploration reward keeps it searching. Training only the actor without rewarding exploration causes the score to drop.

The construction of the training task stream also matters. For the same 6+6 stream, ordering from easy to hard gives +2.27 cumulative gain, shuffling the order gives +0.69, and shrinking each segment to 3 tasks gives +0.58. The original explanation is that early tasks need to be easy enough for the model to learn how the repository is organized and how tests are run, so that later hard tasks have a chance to use memory. If the training stream is too short, memory does not have enough opportunities to be used, and the writer learns little worth retaining. This detail is useful for people building agent products: giving a memory file is only the starting point; task order and length change the value of memory. On two held-out long streams of 26 and 27 issues, the model trained with the 6+6 curriculum still has the highest cumulative gain.

There is also a branch experiment: shared parameters are better than decoupled parameters. Freezing the writer and training only the actor makes cumulative gain almost zero. Training the writer separately recovers some gain but is still lower than continuing training with shared parameters. The original speculation is that shared parameters let the actor's exploration reward and the writer's improvement reward update the same set of weights, so learning in one role can help the other.

There are several questions in the comments worth noting. A reader asked what would happen if a 4B model wrote the notes and a frontier model solved the tasks. Another suggested giving the writer a fixed storage budget and rewarding downstream success; delayed revisiting might help it judge which experiences are worth keeping. Someone else asked about forgetting: when an agent turns memory into small note files, the hard part is deciding what to delete. Another comment mentioned the reward horizon: notes that help with the next task and knowledge that is still useful a month later may not be the same thing. Someone asked whether the writer can recover from its own erroneous notes. The original paper does not answer all of these questions, and the article also lists future work: efficiency, integration into agent systems, how to train when experiences are not a clear task stream, the trade-off between writing memory and updating parameters, and whether a similar exploration reward can be used when multiple agents work in parallel.

Finally, back to that scenario. For agent memory, the storage system is only the surface. What is really hard is training the judgment: what in this experience is worth writing, and what would harm the next task if written. The signal RIME provides is concrete: score memory by the performance of subsequent tasks, and then let the actor explore in order to write useful memory. The fact that a 4B model can match the cumulative reward of frontier models shows that this path at least works at small scale. But the original text also admits that the scale is still small, and efficiency and integration are open questions. For people building agents, don't rush to pile up memory files first; see whether your task stream gives memory a chance to be used.

Original paper and project page: https://cmu-aire.github.io/RIME/ , leaderboard: https://continual-learning-bench.com/leaderboard.html