Agent self-evolution gained 13.5 points; with a different model, only 2.2 remain

Agent self-evolution has seen many methods over the past two years: some modify the prompt, some the skill, some the tool, and some directly modify the harness—the execution and orchestration layer outside the Agent. The problem is not a shortage of methods, but that comparison is hard. Many papers build their own execution environments, evaluation methods, and search processes; scores are all rising, but the underlying conditions differ. It is hard to judge whether the improvement comes from the method itself or from a more convenient experimental setup.

RSIHub aims to solve exactly this. The paper's authors made it an open-source framework that unifies the execution environment, evaluator, data splitting, version management, and recovery mechanisms. Different evolution methods are only responsible for deciding: which candidate to select, how to view trajectories, what to modify, and what to keep. The authors brought AHE, Hyperagents, A-Evolve, and GEPA into the same framework without modifying the framework core; method-specific logic stays in recipes and operators.

Experiments were run on Terminal Bench 2 and τ³ Banking, using MiniSWE and Codex, with model and evaluator configurations fixed. The results are concrete: on Terminal Bench 2, MiniSWE improves from 56.2% to 69.7% at best, a gain of 13.5 percentage points. Switching to the stronger Codex, the maximum improvement is only 2.2 percentage points. On τ³ Banking, method rankings change again. Some methods improve their overall scores but regress on the held-out split. The held-out split can be understood as the portion of tasks set aside and not used for tuning.

This is also the point that a commenter named EVE emphasized in a reply: if a method only gets better in the environment it was tuned on, then it is overfitting to the harness, not truly learning. A public testbed is long overdue. Another commenter said bluntly that inconsistent evaluation environments are painful.

RSIHub's value lies more in making candidate histories inspectable, binding evaluation conditions to results, and making components reusable—rather than creating yet another leaderboard. People are already very good at getting Agents to modify themselves. What needs to be judged next is whether it has truly learned, or has merely tuned itself more smoothly to the current environment.

Paper page: arXiv:2610.32253
PDF: arxiv.org/pdf/2610.32253
HTML: arxiv.org/html/2610.32253v1