Claude Opus Treats Hyperparameter Tuning as a New Discovery; Three Skills Lift Improvement from 10.4% to 20.1%

There is a detail in a new Anthropic paper: when Claude Opus does scientific research, it treats a hyperparameter-tuning trick that almost everyone uses as a creative breakthrough.

The paper is titled Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?, and the authors are Ming Zhong, Stacy Li, Nicholas Carlini, and Matthew Jagielski, from the Anthropic Fellows Program and Anthropic; it is listed in the NeurIPS 2026 E&D Track.

They built Agent4Sci, a long-horizon testing framework. They put Claude Code, Codex, and Gemini into 20 real ML for Science benchmarks spanning 11 domains. The task is straightforward: read papers, modify code, run experiments, and try to beat the papers' original best results. It does not rank a leaderboard; the focus is on what today's agents can achieve under in-the-wild conditions, and where they get stuck.

The results are not bad. Frontier agents found improvements on all benchmarks, with Claude Opus achieving an average relative improvement of 10.4%, and no run was flagged for academic misconduct.

But after breaking down the trajectories, the authors found that most of the gains came from familiar ML engineering tricks: tuning hyperparameters, switching optimizers, ensembling, and repeated local search. Genuine scientific judgment is still relatively weak. Three limitations recur in the trajectories: overestimating the novelty of an idea, leaving GPUs unused, and, after quickly finding a promising direction, continuing to tune parameters around it.

One online commenter said this is like a first-year master's student who has just joined a lab, treating getting default parameters to run as rewriting the history of the field. Someone else added that it is like an intern who added a cache and wrote it up in the weekly report as an architectural innovation. A reply put it even more directly: the reward function only measures how much the loss dropped, not whether the trick was already described in a 2018 paper.

The authors then added only three lightweight Skills: research ideation, experiment management, and periodic self-checking. Claude Opus's average improvement rose from 10.4% to 20.1%, and Codex's from 4.4% to 8.1%; the interventions also became more diverse and more domain-appropriate. Experiment execution is already strong enough; what holds it back is judgment. Rather than continuing to pile on compute, it is more cost-effective to first give it scientific research habits.

Some commenters suggested splitting the evaluation table into two columns: how much the metric improved, and what is new relative to existing methods. The former lets results speak, while the latter returns to the literature and controls. Others said to add an external check before an agent claims a breakthrough: state which existing method this trick resembles and which baseline it beat; if it cannot answer, it should count only as a hyperparameter-tuning run and not go on the discovery list. There is also a simpler approach: first have the agent compare against the three closest methods in the literature and clearly explain where the differences lie.

If you often use Claude Code or Codex to run experiments, this paper is worth reading.

Paper: https://arxiv.org/abs/2610.28763

Code: https://github.com/maszhongming/agent4sci

Data: https://huggingface.co/datasets/MingZhong/agent4sci