All AI Labs Are Taking the Same Exam, It's Just That No One Mentions How Many Retakes There Were

Jason Wei did something very nerdy: he counted every benchmark that frontier labs reported in their release materials in 2026 and turned it into a frequency distribution chart called 'the benchmark of benchmarks.'

The result looks like this.

Four exam papers are almost universal: HLE, SWE-Bench Pro, DeepSWE, and Terminal-Bench were all reported by all 12 frontier labs. Simply put, HLE tests expert-level knowledge Q&A, the SWE-Bench series tests code editing, Terminal-Bench tests working in the command line, and DeepSWE is also a software engineering task. Scale AI was involved in two of these four, SWE-Bench Pro and HLE in collaboration with CAIS; MCP-Atlas ranked ninth.

OpenAI has authored the most tests: SWE-bench Verified, BrowseComp, and GDPval. Using tests made by competitors seems to pose no psychological barrier to anyone.

Anthropic has only one on the list, BioMysteryBench, which basically no one uses. In the comments, some were anxious on its behalf, while others thought this precisely shows that the list measures release habits, not capability.

Academia contributed quite a bit. Princeton has two, SWE-bench and CharXiv; Stanford has one, Terminal-Bench; Berkeley has two, CyberGym and Agents' Last Exam; NYU has one, GPQA.

DeepSWE deserves a separate mention: it first appeared only in June, and by the end of the year all 12 labs had reported it. Terminal-Bench is even more extreme: within one year it iterated from 2.0 to 4.0, four versions.

Looking at the chart, it's easy to think the industry has finally reached a consensus. But the term "universal adoption" doesn't hold up under scrutiny.

First, the filter conditions. Some commenters pointed out that the chart only includes benchmarks used by at least three companies in their release materials. In other words, adoption rates are measured within an already filtered pool. Niche but discriminative tests were never on the chart to begin with.

Then there are versions. With four versions of Terminal-Bench in one year, different labs may have reported four different tests. Cloris said in a reply that adoption rates by benchmark family are fine, but scores must be precise down to version and harness; otherwise "sharing a name" does not equal "sharing an exam."

Then there are attempts. Kritin Vongthongsri mentioned their rerun results on τ-bench: 83% passed on the first attempt, but when run 8 times in a row, only about half passed all eight. Release materials usually report only the first-attempt number.

Finally, the definition of adoption. Max Petrusenko asked bluntly: does "adoption" mean a lab wrote a number on a slide, or that an external team can rerun it and get consistent results? On the chart, these two things look exactly the same.

One commenter's summary was even harsher: HLE appearing in 48 reports shows convenience, not rigor. Picking a test that is easy to put into a launch event costs far less than picking one that truly distinguishes capability.

So the chart's real value is not in the ranking. It turns launch-event habits into a curve: four tests being universal shows that the industry has temporarily reached consensus; changing versions four times a year, reporting numbers from a single run, and filtering out small samples show that this consensus is looser than it appears. Jason Wei said he wants to do it once a year; if he really does, it will probably be the meta-evaluation most worth following: watching how the same ruler gets redefined.