The AI Index Has Changed Exam Papers—Can Leaderboard Scores Still Be Compared Directly?

Artificial Analysis Intelligence Index is a model leaderboard that weights multiple benchmarks into one overall score. Hamel Husain pulled the weight history of the current 9 benchmarks and found that the weights have changed considerably over time. The index consists of 9 benchmarks, and the weights are set manually.

Once the weights move, rankings can move. AlphaBeta offered a quantification habit: randomly perturb around the published weights, for example sampling from a Dirichlet distribution, re-rank, and count how often the No. 1 model still holds first place. If the champion flips after a small change in weights, then what this index measures is closer to the weight configuration. Zeeshh asked more directly: are there models that climb the leaderboard purely by redistributing weights while their individual capabilities remain unchanged?

Harrison thinks weights are only a smaller part of the story. According to his count, only 3 of the 10 benchmarks from January are still there, and v5 has 40% of its weight placed on 3 benchmarks added on October 27. v4 scores and v5 scores are basically two different exams. Omkar Ray added that in v5, Briefcase, FrontierCode 2.0, and Terminal-Bench-Science each account for 15%, while HLE and SciCode were retired. Part of a jump between versions comes from the ruler being changed. He asked whether anyone had re-scored old models using the v5 weights. No one in the material provided a result.

This is more troublesome than weight drift. Weight changes can still be recalculated, but once the exam paper changes, old scores lose the same yardstick. Simon Livson pushed the question to the extreme: would today's leading model still lead if placed into the frozen v4 weight combination? Aniket Tapre suggested putting the weight history directly next to the scores, so that at least readers know where the largest shift occurred.

In a reply, Mert Gulsun explained the update logic. Their goal is to give these systems, whose capabilities change quickly, a balanced and up-to-date view, so the index components need continuous monitoring and improvement. Reasons for updates include test saturation and shifts in user and vendor use cases. He also said the index should be seen as the starting point of a deeper journey; they have also made many other indexes and benchmarks, as well as Optima, which lets users build their own benchmarks. This response did not directly answer the question of re-scoring old models, but it made clear that the index will keep changing.

Others looked at individual items. Tim Lai said he wants to see a column for each benchmark, not just the overall score. The neutrality of a weighted average depends on who is choosing the weights. akshit.io noticed that in v5 the knowledge work weight dropped from 40% to 25%, and category weights returned to 25% each. Eddie gave a practical suggestion: if you use the index to choose models, note which version of the weights you are looking at; a headline score a month later may not be directly comparable.

Yoeven raised another detail: on GDP.pdf, Artificial Analysis first uses another OCR model to extract text, then hands it to the provider being evaluated. He questioned that native document processing should originally be part of the benchmark; some models have stronger OCR, and unifying OCR first may change output quality and also make this test deviate from the original GDP.pdf. He posted a screenshot.

Putting these replies together, two kinds of uncertainty emerge. One is that weights are moving, and models can rise through changes in the weighting mix. The other is that the questions are changing, and total scores between versions may not measure the same capability. For ordinary readers, the index can still quickly give a rough position. For people using it for model selection, individual scores, weight versions, and the question list may be more useful than a single total score. Next time you see a model's score rise by some amount, first ask: which version of the exam paper is this, and which weighting mix?

Original tweet: https://x.com/HamelHusain/status/2120340744665989528