Wink Pings

Meta Uses Agents to Automatically Build Benchmarks from a Codebase, with Quality Checks Requiring No Per-Task Human Annotation

Building coding evals by hand is expensive and prone to bias. Meta's Benchmark Factory lets agents automatically generate evaluations from merged code changes, relies on quality anchors that require no per-task human annotation, and deliberately feeds in bad benchmarks to validate their effectiveness.

Meta Uses Agents to Automatically Build Benchmarks from a Codebase, with Quality Checks Requiring No Per-Task Human Annotation

Evaluating coding agents requires a set of tasks and scoring criteria, that is, a benchmark. Off-the-shelf benchmarks may not fit a team's actual context, and building and validating them by hand is costly. Meta's new paper proposes an agent loop called Benchmark Factory that automatically builds coding evals from merged code changes and automatically checks them with a set of quality anchors that require no per-task human annotation.

Benchmark Factory

How the Loop Works

Merged production code changes in a repository naturally come with a 'requirement' and a 'correct result'. An LLM agent first writes a sourcing query to select tasks from these changes; then it runs a set of models with known relative strengths on the candidate tasks. If the results are wrong, the agent diagnoses the failure, modifies the query, prompt, or configuration, and keeps fixing. This continues until the benchmark passes the quality anchors.

Quality anchors are automated checks that do not require per-task human annotation. For example: a more capable model should score higher, and the gap should be sufficiently clear; difficulty should fall in a reasonable range; variance across multiple runs should not be too large. The scorer and expected ranking are frozen before starting, so the agent cannot bluff its way through. Each repair round uses fresh context, with state written to files. If it still fails after reaching the round limit, it is escalated to a human.

How They Prove the Checks Work

The authors deliberately create bad benchmarks and feed them in. Benchmarks that are too easy are rejected; five types of bad scorers are rejected; when model labels are shuffled and no true ranking exists, at most only 1.4% of randomly resampled bad benchmarks slip through. On held-out configurations that were never involved in construction, these benchmarks can rank Claude Opus 4.7 with high reasoning effort above low reasoning effort (60.6% vs 54.4%, p=0.002), and GPT-5.5 above GPT-5.4 (65.8% vs 60.5%, p=0.004). 9/10 team-level comparisons agree in direction.

The paper notes that quality anchors are a necessary but not sufficient condition. Engineers rated a 5% sample, with task quality averaging 2.3 points and grading judgments averaging 2.6 points out of 3. Human review is still needed, but it can focus on areas the loop cannot fix.

Comments

One commenter asked under the tweet whether the authors first strip the fixes out of git history, since agents are very good at seeing answers in git log. Another commenter thinks generating tasks from merged changes is clean: the repository history itself knows what 'done' looks like. Others worry that the 'strong models should beat weak models' anchor may filter out tasks where smaller models are better at internal conventions—exactly the kind of tasks teams often care most about. There was also a suggestion to regenerate every few months; otherwise, the benchmark will remain stuck at 'the code state when it was built' and fail to keep up with an evolving codebase.

Links

发布时间: 2026-11-20 22:37