AI Reviews and Improves Its Own Work After Finishing—Only 2 of 7 Tests Actually Solved the Problem

After a coding task is done, let the AI look back at the work it just completed, list a few revision suggestions, and let a human decide whether to adopt them. This is what skills like /retro do. There are many similar self-evolution mechanisms: after the AI finishes work, it looks at where it got stuck, proposes changes to Skill files, and avoids taking the same detour next time. No model retraining is needed; just change configuration and documentation.

Sounds great. But some have long suspected that without clear criteria, self-optimization is hard to make reliable. Relying entirely on human review is also tiring: it demands a lot from people, and reviewing every time is unrealistic.

A paper specifically tested this. “How Do Coding-Agent Harnesses Self-Evolve? An Empirical Study of Retrospective Adaptation,” by Christoph Treude of Singapore Management University and Sebastian Baltes of Heidelberg University in Germany, was published on arXiv on October 26.

How it was tested

The study selected 30 problems from SWE-bench Verified, code-repair tasks collected from real open-source projects, involving Python projects such as Django, SymPy, and Astropy. The first 10 were sampled randomly, and the next 20 were specifically chosen from projects where environmental issues had been encountered before. Claude Code ran /retro after each problem was fixed, producing 125 suggestions in total.

The authors selected 24 suggestions for Claude Code to implement, of which 11 could be tested, plus 1 control written by the authors themselves, for 12 items in total. The test method was to find a task in the same project that was confirmed to trigger the same problem, run it 5 times with the original settings and 5 times with the modified settings, and compare whether the AI still performed the action it was meant to avoid.

The tool was Claude Code, and the model was Opus 5.5.

Four findings

Most suggestions were grounded, but not all. Of the 125 suggestions, 101 addressed things that actually happened during the task. The other 19 had rationales unrelated to the current task, and 18 of those were suggestions to add code-format checks. One reason is that /retro's instructions themselves specify that if a project has no pre-commit checks and no automated process, that counts as a finding. The authors configured one of these—“automatically run format checks before committing”—and tested it; in all 5 runs, the AI never read it or ran it.

In only 7 of 13 tests did the original settings actually hit the target problem. Of those 7, two completely solved the problem.

One was stating in CLAUDE.md that SymPy should use bin/test to run tests and noting that pytest was not installed. Before the change, in all 5 of 5 runs the AI first tried pytest and failed; after the change, 0 times.

The other was writing a “list of tests that are expected to fail anyway” for Astropy. Previously, when the AI encountered a test failure, it would rerun it on unmodified original code for comparison to confirm whether the failure was caused by its own changes. With the list, such comparisons dropped from 3 out of 5 runs to 0.

The remaining 5 changes did not cover the actual scenario of the next task. For example, the failing tests listed did not match the next batch encountered. Of these 5, only 1 reduced comparisons from 5 times to 2; the other 4 showed no change.

Some changes actually increased workload. The failure list for the Sphinx project said, “Any failure not on the list should be treated as caused by your own changes.” As a result, on an unrelated task, in 4 out of 5 runs the AI performed extra comparison checks, up from 0, and the median time rose from 82 seconds to 245 seconds. The earlier effective CLAUDE.md also added the sentence “Run format checks before finishing”; the AI followed it every time, and the median time rose from 88 seconds to 156 seconds.

As for whether the problem could ultimately be fixed, the number of successes in each group of 5 differed by at most 1, so no improvement in success rate was visible.

What can remain

The paper's conclusion is not complicated. Project-wide information—such as which command this project uses to run tests—is useful once recorded and remains useful later. Information such as which tests fail in a particular environment may no longer match when the task changes.

To evaluate a change, look at three things: whether the problem occurred, whether the AI read the change, and whether behavior changed. Retained changes also need continuous evaluation, because outdated or incomplete instructions may not work and may even increase workload.

The authors also acknowledge limitations: each setting was run only 5 times, so the sample is small; the test tasks were deliberately chosen to reproduce the problem, which may overestimate the effect; and only one tool and one version of /retro were tested, covering only environment-related issues.

A more fundamental question

Baoyu made a judgment in a tweet: only Skills that can be automatically accepted programmatically can even be said to self-evolve. For example, optimizing code performance—performance is measurable and quantifiable; give it test cases to iterate on, and it can get better and better. For writing, which has no clear acceptance criteria, AI self-scoring differs considerably from real human perception; a draft the AI scores highly may read to a human as full of AI flavor. If you really want to write a good Skill, humans still need to use it and point out the direction.

A netizen, Hongliang, put the phenomenon more directly: in optimization theory this is called proxy overoptimization, over-optimizing a proxy objective. For tasks like code review, the proxy objective is close to the real objective; for tasks like writing and aesthetics, the real objective is human preference, and AI self-evaluation is only a proxy. If you optimize based on that score, the AI will cater to the criterion of “what AI thinks is good,” and its outputs will become increasingly strange.

Some teams are also trying to make this more engineering-driven, such as SkillOpt from Microsoft and several universities, which uses a training-like framework to evaluate and optimize skills and sets validation gates to control editing direction. The idea is right, but the premise remains unchanged: there must be quantifiable acceptance criteria, otherwise the validation gate itself becomes decoration.

Back to the most practical question. The step where AI reviews and makes suggestions is not scarce; what is scarce is judging which suggestion should stay. Of 125 suggestions, only 2 truly solved the problem—this ratio is not surprising. Without acceptance criteria, retrospection easily becomes self-comfort; with criteria, only some tasks can run automatically. The rest still need people to look.