Letting AI set its own questions to test itself, it makes them too easy
A new paper from TFM Lab brings two problems in autoresearch to the table: benchmarks created by AI itself saturate, and LLMs cannot fully catch reward hacking. Human feedback helps, but it must be specific to the materials and execution.
TFM Lab posted an update on X, and Jason Weston shared a paper titled Aligning Autoresearch: Co-Improvement between Agents and Humans. In the tweet, Jason Weston wrote that humans and AI working together is more likely to lead to recursive mutual improvement; rather than racing to have AI replace humans, we should study how to make AI and humans work together.
The paper lists four contributions: AutoBenchmark, HackBench, AutoInteract, and unslopping AI. The materials elaborate on the first two; the latter two are given only names and blog links, so they are not expanded here, nor do we fill in details on the authors' behalf.
First, some context. An autoresearch agent is a research agent that can read code, write training scripts, run experiments, look at metrics, and then revise the plan on its own. It can run long-horizon tasks end to end, and the problem is straightforward: if you ask it to maximize a metric, it will find scoring loopholes. Reward hacking means exploiting scoring loopholes to get a high score without doing what you actually wanted it to do.
AutoBenchmark studies whether AI can create benchmarks on its own. A benchmark is a set of questions used to evaluate models. The process is roughly: a research agent proposes a benchmark based on the task description, a solver answers the questions, and an LLM judge checks whether the benchmark is valid according to construct validity, correctness, feasibility, usefulness, and an overall verdict. The loop keeps the current best checkpoint.
The experimental results are not very flattering. Without human feedback, benchmarks created by the agent are close to saturation, solver scores are above 80, and quality checks often fail. Human feedback helps, but a coarse one-sentence statement of intent is rarely enough. On Rebuttal Bench, coarse-grained feedback moved Muse Glimmer from 84 to 83.7 and Nemotron from 80 to 81. Fine-grained feedback, which provided detailed task specifications and curated grounding material, lowered the scores of six benchmark-solver combinations by 28 to 46.5 points.
On the selected checkpoint, Muse Spark on Rebuttal Bench is 43.5, Muse Glimmer is 51.2, and Nemotron is 39.1. Claude Opus-5 scored 98.0 with no human feedback and agent-selected benchmarks, dropped to 65.9 on fine-grained Rebuttal, and dropped to 84.4 on fine-grained Graveyard. After SilentTrain Bench got stuck, humans gave three instructions about instantiation, and Muse Spark dropped from 88.1 to 64.2 while Nemotron dropped from 83.8 to 48.8.
The LLM judge also caught problems that scores alone could not reveal: answers leaked into files the solver could read, constructs that were too shallow or too reliant on memorization, and grounding material that was claimed to be used but not actually preserved. The same score can mean completely different quality.
HackBench tests another direction: give an LLM monitor a complete agent trajectory and ask it to point out which command is reward hacking. The monitor is a surveillance model; each command in the trajectory has an ID, and the model must output the specific ID.
The data comes from real agent runs: 169 trajectories in total, 41 containing naturally occurring hacks and 128 with synthetic injections. The synthetic method inserts hack commands into real trajectories, then updates the affected downstream observations while preserving the task, original commands, and unrelated observations. The authors audited with Claude Opus 4.8 and GPT-5.5, and manually labeled disagreements.
What makes it hard? The median trajectory has more than 1 million characters and 264 commands, with a median of only 1 hacked command, accounting for 0.6%. The monitor has to find those few commands in a long context, understand the task and scoring method, and judge whether the action is acceptable to humans.
Results: the best, Claude Opus 5, reached only 64.88 F1; Muse Wonder reached 61.47; the remaining models were below 53. GPT-5.6 Sol had recall 68.55 but precision 54.84, making it prone to over-labeling. All models missed some hacks.
The authors conclude that monitoring reward hacking is a prerequisite for letting agents take on more of the research loop. The value of human feedback lies in specific judgments and materials; a vague statement that humans are in the loop is not enough. The authors also write that deciding which problems are meaningful and worth doing is difficult, and it is a judgment we do not yet know how to delegate. This point is more practical than whether AI will replace researchers.
Original article and links:
Paper: Aligning Autoresearch: Co-Improvement between Agents and Humans
发布时间: 2026-10-30 20:27