Wink Pings

After Switching Models, Harness Loses Points: Google Gives the Optimizer a Mistake Notebook

A new Google paper proposes Self-Improvement Memory, letting the harness optimizer remember the reason for and result of each edit. In ten coding continuation runs, scores improved nine times, averaging +2.36, p=0.003.

A paper from Google Cloud AI, DeepMind, and Penn State pulls out a step in self-improvement that is often skipped: memory.

First, some context. A harness can be understood as the runtime framework wrapped around a model; tool calls, workflows, prompts, and evaluation scripts are all inside it. Recursive self-improvement (RSI) in this paper operates at the harness layer: the optimizer changes the runtime framework and does not touch model weights.

The problem arises when switching models or benchmarks. Harness optimization usually assumes the policy model and evaluation set are fixed. Once either one changes, the optimizer must decide which parts of the old harness to keep and which to change. But the harness itself does not explain why changes were made in the past or how effective they were. The paper gives a very concrete failure number: a harness evolved by Opus, when placed on Gemini, starts at 68.41, lower than the original seed harness at 73.89.

Screenshot of the Self-Improvement Memory paper

The paper proposes Self-Improvement Memory (SIM), installed on the harness optimizer; the task policy does not carry this memory. SIM groups past edits by recurring harness problems, and each edit stores four types of information: the rationale for the change, the predicted result, the observed result, and the evaluation context. New attempts continue to be appended to this history. Both the proposer and selector read it before acting.

Results were compared on four domains, including Terminal-Bench 2.1 through Terminal-Bench 4. In six cross-dataset settings and four Opus-to-Gemini settings, SIM's final score was higher than harness-only continuation and also higher than root-cause-only continuation. In ten independent coding-source runs, paired continuation improved nine times, averaging 2.36 points higher than harness-only continuation, p=0.003.

When a user on X reposted this paper, they reminded people to bookmark it. The authors' advice is very engineering-oriented: if you are automatically evolving a harness, keep the rationale and results of each edit together with the code.

Engineers are familiar with this kind of thing. Logs contain only diffs, without motivations or results, so after switching models the optimizer can only redo trial and error. The value of SIM is that it turns edit history into readable context, so the next modification does not have to guess from scratch.

Paper link: https://arxiv.org/abs/2611.18856

Chat with Paper: https://academy.dair.ai/papers/self-improvement-memory-enables-continual-recursive-self-improvement-2611.18856

发布时间: 2026-11-18 09:38