The Most Misunderstood AI Capability Chart Gets an Update: Coding Ability Doubles in 4 Months, Judgment Tasks Still Failing
Nonprofit AI evaluator METR has updated its widely recognized industry capability time-span chart. The trend of frontier AI models’ ability doubling every four months on verifiable tasks persists, but top-tier models still haven’t reached human-usable standards for research tasks involving judgment and open-ended design, and existing data cannot be used to project the long-term iteration pace of such capabilities.
Nonprofit AI assessment organization METR released its Q4 2026 Capability Update Report, upgrading the widely circulated (and heavily misunderstood) AI capability time-span chart. For context on these misinterpretations, refer to a prior report.
This chart follows a straightforward calculation logic: METR first prepared a set of tasks completed by humans, graded by the time humans took to finish them. It then converted the maximum difficulty task a model could complete into the corresponding human completion time, using that as the model’s capability metric. Older versions of this metric showed that frontier models could successfully complete tasks equivalent to double the human time, a doubling every four months.
Many have used this growth rate to directly project the arrival of artificial general intelligence, ignoring the old evaluation’s scope. It only included verifiable tasks—like programming or math problems—where correctness can be automatically checked by software. Researchers train models with reinforcement learning, repeatedly practicing problems, reinforcing neural pathways for correct answers and weakening those for wrong ones, which delivers steady score gains. This is why AI progresses so rapidly in programming and math.
The old evaluation entirely omitted tasks without clear correctness standards, which require subjective judgment—exactly the core capabilities AI needs to take over R&D work, and even achieve self-iteration.
This new evaluation specifically adds two task categories with strong judgment requirements, directly targeting AI’s core need for automated R&D.
The first category is AI evaluation design: METR asked models to create an AI evaluation test that METR employees could directly use, then scored it manually. The results showed a stark contrast: current top models’ software engineering skills already outpace most human professionals, yet none of their produced evaluation schemes meet a deployable standard. The best performer, Opus 5.5, only barely scraped the passing mark. A year earlier, all tested models failed this task entirely.
The second category is philosophical argument scoring, which tests models’ general judgment. Their performance here also falls short of human levels. METR internally debates the relevance of this task to AI R&D.
OpenAI’s recent official blog notes a similar gap: its own models perform worse on real employee assignment tasks than on simple verifiable tasks in METR’s old evaluation.
Earlier this month, a heavily redacted METR report leaked, claiming internal Anthropic AI had doubled R&D efficiency, sparking widespread discussion of recursive self-improvement. Many fear AI will soon operate without human involvement, independently designing and iterating next-generation models. This new report’s results add a critical reference point to that debate.
Scholars disagree on the performance gap between these two task types.
Sayash Kapoor, a UC Berkeley computer scientist set to join the faculty, isn’t surprised by these results. His August preprint paper notes that AI cannot yet independently replicate human computer science research. He argues that research taste—choosing research directions and screening valuable hypotheses—will be the core bottleneck for automated AI R&D.
Another group holds the opposite view. Anthropic CEO Dario Amodei and other researchers, in public interviews and industry articles, argue that if training on verifiable tasks is sufficient, model capabilities will naturally transfer to open-ended tasks. The progress on verifiable tasks shows no signs of slowing.
A bigger problem is that evaluation of these judgment tasks is still very basic, so it cannot produce clear growth curves like verifiable tasks.
The most immediate constraint is cost. High-difficulty tasks may take hundreds of hours for human experts to complete; hiring large numbers of people to test them is prohibitively expensive in both time and money. Most human completion times for new report tasks are estimates, not based on real human testing.
Task design also has flaws. For example, in the philosophical argument scoring task, most samples only have one rater, so model scores essentially align with one person’s judgment. If scored by a different set of experts, results could shift significantly.
METR technologist Matt Putz noted in an email reply that hard-to-verify, judgment-heavy tasks are inherently difficult to quantify, and their trend lines won’t be as clear and stable as standardized verifiable tasks. Boston University computer scientist and linguist Najoung Kim also said the observed small gains only reflect current conditions, and cannot be directly used to project future iteration speeds.
All interviewed researchers support METR’s direction in expanding evaluation boundaries. Sayash Kapoor says more innovation is needed in AI evaluation at this stage, and moving toward open-ended tasks is a good start. But everyone acknowledges current data and evaluation methods are far from sufficient, no one can give a clear timeline for automated AI R&D, and related discussions will continue for a long time.
发布时间: 2026-11-21 02:15