Four mainstream AI benchmarks all labeled 'flawed' by an independent third party

Hamel Husain posted on X, saying the more he looked at public benchmarks, the more shocked he became: the industry relies on them so heavily, yet their quality is absurdly poor. He mentioned that xeophon recommended Epoch AI's benchmarks site. Epoch AI is an independent third party that specializes in reviewing benchmarks, and one of its methods is 'looking at the data'.

The image shows review conclusions for four common benchmarks; the Epoch verdict column is all Flawed. Terminal-Bench 4.0.0, DeepSWE v1.1, OSWorld 2.1, and HealthBench Professional all failed to escape. Below the table are their respective findings: task errors, scoring bugs, structural problems. Hamel's advice is direct: before trusting a benchmark, it is worth running it yourself, carefully looking at examples where the grader made a wrong judgment, and getting a feel for the quality.

Benchmarks are like exam papers, and graders are like the teachers grading them. If the exam paper has wrong questions and the grader is also fixated on format, scores become distorted. Someone in the replies put it more harshly: the grader is usually the worst part; half of the 'wrong answers' are regexes blocking correct answers. Another reply mentioned that sorting grader failures by model confidence and reading the first 50 is a cheap way to estimate a benchmark's noise floor. This suggestion came from Bali, who mentioned Kaggle experience: when the best model confidently gets a sample 'wrong,' it is often the label that is wrong.

There are also many concrete examples. One user said that in HealthBench Professional, 25 of 50 tasks have clear errors; this number appearing in a launch post that people cite is crazy. Another said Terminal-Bench 4.0 has 30/66 tasks broken, and kimi 65.7 versus opus 66.4 is basically a coin flip, not a ranking. MMLU was also brought up: MMLU-Redux manually checked 5,700 questions and estimated that 6.49% had errors, with the virology subset at 57%. These numbers come from replies, not the original conclusions in Epoch's table, but the direction is consistent.

Epoch's review is not without rebuttals. Terminal-Bench contributor Ryan Marten said he had been excited about this initiative but was a bit disappointed by the rigor. He cited Yafah Edelman's audit, saying Terminal Bench 4.0 had 45.5% of tasks broken, 46% of 48 random HLE samples broken, and DeepSWE 1.1 had a bug that could break scoring for every task. Epoch AI's Alexander Barry later responded that Yafah was not involved in the TextQuests review, did not fully understand the nature of the complaints, and apologized for the confusion caused. Another reply pointed out that Epoch did not discover these problems itself but compiled them from Terminal-Bench's public issue tracker, many of which were submitted by bespoke and TB people. Benchmark reviews themselves also need to be reviewed; this loop is quite interesting.

Hamel said he is still collecting data and will write an article later. For ordinary users, remember one action: when you see 'some model improved by X points on some benchmark,' do not rush to repost. Randomly open 20 questions and read them, or look at how the grader scores. For developers, one reply said it accurately: treat benchmark code as part of evaluation, check the grader, not just the model.

The industry uses these numbers for launches, fundraising, and roadmaps. If the exam paper itself is flawed, the prettier the scores, the more it looks like testing who is better at dealing with the grader.