Wink Pings

A harness tuned on Opus scores only 68.41 on Gemini: what self-improvement lacks is an edit memory

This Google RSI paper puts memory on the harness optimizer. A harness evolved on Opus scores only 68.41 when switched to Gemini, below the seed harness's 73.89. SIM stores the rationale, prediction, and outcome of each edit; it wins 9 out of 10 coding paired runs, averaging 2.36 points more. The caveats in the comments are also worth reading: only the coding results were replicated, and run-to-run noise and the cost column may skew the conclusions.

Google just released a paper on RSI. The tweet says it's worth bookmarking, with the focus on the harness layer. A harness can be understood as the workbench outside the model: tool calls, context organization, the evaluation loop, prompts, and patching logic all live in it. RSI is often translated as recursive self-improvement. This paper grounds it at the harness layer; the model weights stay fixed, and what changes is the workbench outside.

Tweet screenshot

The paper is titled Self-Improvement Memory Enables Continual Recursive Self-Improvement, arXiv ID 2611.18856. It studies a very practical problem: harness optimization usually fixes the model and benchmark. Once the model or benchmark changes, the optimizer has to decide which parts of the old harness to keep and which to change. The problem is that the harness itself does not explain why it was changed in the first place, nor does it say how well the change worked.

The paper gives a striking data point: a workspace harness evolved on Opus starts at 68.41 when switched to Gemini, lower than the ordinary seed harness's 73.89. In other words, the tuned version starts out behind on the new model. Commenter Fouad says an evolved harness without an edit history is roughly someone else's dotfiles. The analogy isn't flattering, but it's accurate.

The method is called Self-Improvement Memory, or SIM for short. It is installed on the harness optimizer, not on the task-solving policy. SIM groups historical edits by recurring harness problems. Each edit stores four kinds of information: why it was changed, what was predicted to happen, what actually happened, and what the evaluation context was. Both the proposer and the selector read this memory before making new changes. New attempts keep getting appended to it.

Paper first page

Results: across four domains, including Terminal-Bench 2.1 through Terminal-Bench 4, SIM scores higher on six cross-dataset and four Opus-to-Gemini settings than simply continuing to use the harness or continuing only root-cause analysis. In paired continuation experiments with ten independent coding source runs, SIM won 9 times, with an average improvement of 2.36 points, p=0.003.

The result is appealing, but the caveats in the comments cannot be skipped. Beshoy Armanios points out that only the coding results were replicated. Across ten paired source runs, SIM won 9 times, with a mean of 2.36 points, a 95% confidence interval of 1.05 to 3.68, and TB4 scores around 19 to 22. For other domains there was one run per condition, and the authors also say the rollouts did not capture optimization variance. John Rood added a jab: harness choice alone can make the cost per pass vary by up to 4.4x, while pass rate changes by less than 0.4 points. A loop that looks only at accuracy may spend a lot of money buying noise. Cost needs to be stored together with the rationale; otherwise the optimizer will optimize the wrong column.

Reply screenshot

On run-to-run noise, someone linked another paper, arXiv:2611.19359. That paper ran paired tests on a Gemma 4 31B coding agent, and 19% of tasks flipped between two identical no-memory runs. A memory reminder that read +8.3 points on 48 tasks became -10.0 points on the next 50 tasks. This warning is not aimed at SIM, but it shows that memory conclusions from small-sample single runs can easily be sitting on noise. When looking at SIM's 2.36 points, also look at its paired design and confidence interval.

There are a few details in SIM worth pulling out. The memory serves the part of the system that changes the harness, not the agent executing the task. Predicted outcomes are written down before testing and cannot have the story quietly rewritten afterward. Failed edits must also be kept, along with the conditions that made them fail; otherwise the next optimizer will rediscover the same tempting mistake. Aiden asks a good question: if the proposer and selector read the same memory, what stops them from agreeing on the same wrong lesson together? This is probably a verification mechanism that needs to be added later.

For people building agents, the conclusion is plain: if you are going to automatically evolve a harness, keep the rationale and outcome of each edit together with the code. Storing only the diff is not enough, and storing only the final score is not enough either. The log should at least contain why it was changed, what was predicted at the time, what actually happened, and on which model and benchmark it happened. Someone in the comments already plans to ship a version on Monday that records every harness edit. That's fast action; hopefully the log columns won't just say score.

Original link: https://x.com/omarsar0/status/2122701047118635485
Paper: https://arxiv.org/abs/2611.18856
Chat with Paper: https://academy.dair.ai/papers/self-improvement-memory-enables-continual-recursive-self-improvement-2611.18856
Another noise paper: https://arxiv.org/abs/2611.19359

发布时间: 2026-11-18 03:32