Two full rewrites, only +2.8 points, and the error bars still cross zero

ARC-AGI-3 is an interactive benchmark where an agent must explore unfamiliar environments, figure out goals on its own, and learn while playing. On this benchmark, human-guided systems have already achieved near-perfect scores on public games, showing that good designs exist. The question is whether frontier models can find such designs on their own.

First, let's explain what a harness is. It is not the model itself; it is a symbolic program layer responsible for orchestrating various calls for the model: perception, memory, tool use, error recovery, and allocation of reasoning resources. You can think of it as the agent's skeleton and scheduling center. The same model, wrapped in different harnesses, can perform very differently.

The authors are Jens Lehmann (Amazon, TU Dresden) and Sahar Vahdati (Leibniz University Hannover, TIB). The experimental protocol is fixed: each model first uses a fixed coding bootstrap to write a complete task harness; then the harness is actually run on 5 development games; after running, the entire codebase is rewritten twice. Each version is scored on 20 games, and these 20 games never affect any rewrite from beginning to end.

The metric is called RHAE, Relative Human Action Efficiency, which combines level completion and action efficiency relative to a human baseline. Inference cost in dollars is also recorded.

Across 27 lineages, initial harness performance ranged widely, from zero to near-perfect. But after two rewrites, held-out RHAE rose by only 2.80 points on average, with a 95% interval of [-1.44, 7.35], crossing zero.

In other words, statistically it cannot be ruled out that a substantial portion of those 2.8 points is just noise.

A few notable details:

Progress on the development games overstates held-out gains. This is easy to understand: the model repeatedly tunes on 5 games, so of course it gets better and better at those 5. A commenter (@Naol g) said this is basically overfitting in a different skin.

Also, versions that reduce inference cost often lower task scores at the same time. Saving money and saving performance are often a trade-off in this experimental setup.

Human-guided systems still score higher on the same 20 games. This shows there is room for improvement; the model just cannot yet find those better designs on its own.

A commenter (@Ricardo Dias) raised a perspective: initial harness performance ranged from zero to near-perfect, a spread far larger than the gain from two rewrites. If the same model ran the initial construction multiple times and you picked the best one, it might be more cost-effective than one construction followed by two rewrites.

@George pointed out a more fundamental issue: when a model grades itself, it naturally likes its own changes. Corrections that can truly survive held-out evaluation often come from humans reading ugly failure logs that the model cannot even see.

@Champ is concerned with the safety boundary: if a model is allowed to rewrite its own harness, permission checks and output filters cannot be placed in the part it can edit. Otherwise, a rewrite with a higher score might quietly delete some safety check. He suggests diffing each new harness and checking which checks were removed, rather than only comparing scores.

The paper's conclusion is rather restrained: under the two-rewrite protocol, there is a gap between autonomously building a harness and reliably adapting it. Even if task performance is good, it does not guarantee that subsequent rewrites will generalize or preserve existing capabilities.

A more direct translation: models are currently good at making code look cleaner, but not necessarily better at finding genuinely better designs. Two full rewrites may buy nothing more than a prettier commit history.