Wink Pings

Switched models, and the automatically evolving harness actually got worse? Google says the problem isn't the harness itself

A new Google paper focuses on the transfer problem in harness optimization: a harness evolved on Opus drops from 73.89 to 68.41 when moved to Gemini. They gave the optimizer a memory that records the rationale and outcome of each change, and the score improved in 9/10 continued runs.

Google published a paper about recursive self-improvement at the harness level. The authors are from Google Cloud AI, DeepMind, and Penn State. The core problem isn't new, but it has been tested seriously.

Let's clarify the background first. Here harness refers to the scaffolding around the model, including prompt structure, tool definitions, execution flow, memory systems, and so on. Harness optimization means automatically modifying this shell, not the model weights. Past work in this area basically assumed the policy model and benchmark were fixed. The paper pushes the scenario one step further, calling it Continual Recursive Self-Improvement: an existing harness optimization runs halfway, then the benchmark or model is switched, and it continues running.

The real problem is here. The paper gives a concrete number: a workspace harness evolved on Opus starts at only 68.41 when switched to Gemini, while the unoptimized seed harness is 73.89. The evolved result actually holds it back. The reason isn't complicated: the harness itself only records what was changed, not why it was changed, what was expected at the time, and what actually happened. When you switch to another model, the previous set of crutches may not match the new model's weaknesses at all.

Google RSI 论文

Their proposed solution is called Self-Improvement Memory, or SIM. The memory is attached to the optimizer, not to the policy that solves tasks. SIM groups past edits by recurring harness problems, and each record contains four things: the reason for the change, the predicted result, the actually observed result, and the evaluation context at the time. The proposer and selector both read this history before making the next move. New attempts continue to be appended, and the memory grows along with the optimization process.

The experiments cover four domains, including Terminal-Bench 2.1 through Terminal-Bench 4. Under matched starting harness, rounds, and candidate budgets, full SIM beat control groups using only harness continuation and only root-cause continuation in six cross-dataset settings and four Opus-to-Gemini settings. On coding tasks, they ran ten independent source runs; SIM improved the score in 9/10 continued runs, averaging 2.36 points higher than harness-only continuation (p=0.003).

One reader's summary after reading is very direct: if you are automatically evolving a harness, store the rationale and result of each edit together with the code. The harness itself is not the asset; the modification history with explanations is. When switching models, what transfers is not the pile of prompts, but the judgment of why it was changed that way at the time.

Paper link: arXiv:2611.18856

DAIR.AI's explainer page: Self-Improvement Memory Enables Continual Recursive Self-Improvement

发布时间: 2026-11-18 12:26