Memory files might just be a crutch: three Agents did not learn, but a small 9B model learned by changing weights

These Continual-ARC results are not very flattering to agent memory. Daniel Lenton at unify released a new benchmark specifically to test continual learning: whether a model or harness can remember useful things when tasks recur, and apply what it has learned to new unseen tasks. The benchmark uses 25 tasks from ARC-AGI-1, each returning 8 times in a 200-visit stream, with a new grid each time, so caching answers is useless. After the stream, there are two frozen tests: new grids for seen tasks, testing memory; and 100 unseen tasks, one attempt each, testing generalization. The metric is called share learned; it compares the learner with a version of itself with memory cleared, so stronger or weaker models do not get an advantage.

The task page lists all 25 stream tasks and 100 held-out tasks. Each stream task appears 8 times; held-out tasks appear only once at the end.

Three agent harnesses were run on GPT-6 Luna. Daniel's conclusion is blunt: OpenClaw, Hermes Agent, and Prime Agent can solve 93% to 95% of the problems under high effort, but when the same puzzle appears again, they do not get cheaper. Delete the memory, and the results do not change. A baseline that saves the program for each problem, by contrast, lowers cost right away and keeps it down.

More awkward is what happens after they are explicitly told what to remember. Give them a verified program, one per puzzle, and all three harnesses can remember it, saving 17% to 62% of what perfect memory would save. But on unseen new problems, none of them improved. Daniel writes: memory is there; judgment is not. The reasons are in the replies: Prime Agent did not store anything for later at all, and its Python skills failed to import; Hermes Agent opened the skill 524 times out of 525 returns, but mostly as generic hints; OpenClaw saved every conversation but rarely went back to look.

The control group is training weights. Qwen3.5-9B inside Prime Agent, with GLM-5.3 Flash as teacher and expert guidance for DAgger imitation, with two concurrent users, improved from 22.2% to 79.8% on the seen test and from 18.4% to 62.3% on the held-out test. After the small 9B model changed its weights, both memory and generalization improved. In the quoted tweet, Daniel says that training the weights of a small 9B model accomplished both things. A commenter in the replies said that memory files might just be a crutch. Memory files may just be a crutch; what is really hard is getting the model to update its judgment within a task stream.

Daniel also left a caveat: the test covered only one domain and a few models, and they were not the strongest models; stronger models might retain more. But his judgment is that real on-the-job learning requires weight changes, and the harness will become something that trains the model, not an ever-growing pile of markdown files. Beyond controlling the model and giving tools context, the harness is increasingly becoming part of the training pipeline. It can stay small and efficient.

A commenter in the replies made an analogy: one-off tasks are easy; a weird project that started three weeks ago needs notes, receipts, and a map. Memory is the whole task. The analogy fits agents well. Continual-ARC is open for submissions; after running the public stream, you can submit for verification, and unify will rerun the code on 50 externally unseen puzzles. The benchmark does not aim for a single ranking; it wants to explain how learners learn. Both the paper and submission entry are here: paper, submission.