Anthropic's AI Models Are Improving Twice as Fast as They Did a Year Ago

Anthropic is leveraging AI to build more powerful AI. A report published by METR on Thursday put forward a concrete figure: the company's model improvement speed is roughly twice as fast as it was a year ago, with barely any sign of slowdown on the horizon.

Photo: Jason Henry/Bloomberg via Getty Images

This report comes from METR, an AI safety research organization. The on-site investigation inside Anthropic lasted 11 weeks, as part of its work to assess the company's AI research and development operations. The Verge notes that this set of findings provides one of the clearest pieces of evidence we have seen to date on accelerating AI progress.

It is critical to distinguish this work from a separate initiative. METR also runs a separate program that grants safety researchers "employee-like" system access, a policy that was rolled out in the wake of multiple high-profile hacking incidents earlier this summer. These two projects are completely independent of each other. Clarifying this boundary matters a lot, otherwise readers may easily misinterpret the report as METR conducting an investigation into those earlier security breaches.

My assessment: If model improvement speeds double on a year-over-year basis, safety evaluations will become increasingly unsustainable. Evaluation cycles are usually measured in months, and teams have to constantly play catch-up with new model versions, newly emerged capabilities, and newly identified failure modes. The faster the iteration speed goes, the more this constant catch-up work turns into a draining, brute-force grind. For readers with a software development background, this sends a clear message: don't just fixate on new model launches, pay close attention to evaluation methodologies and the cost of reproducing testing results. For regular end users, the short-term experience will likely only feel like more frequent product updates, but the entire underlying R&D loop has already sped up dramatically.

Original source: The Verge. Full report: METR.