OpenAI Pushed Out of the Top Five on the AA Leaderboard: Why Nathan Lambert Focuses Only on a Two-Point Gap

Kimi K3.1 Pro scored 55 on Artificial Analysis, and fawn screenshot a leaderboard chart and said OpenAI is no longer in the top five. Leaderboards change quickly, but Nathan Lambert's focus is not on rankings. He wrote that at the top end of models, real-world usability matters far more than plus or minus 2 points on AA. AA is more like a potential score: some models hit, while others lean toward a miss, and Opus 5 and 5.5 are examples; he thinks K3.1 may be closer to the miss side.

AA is short for Artificial Analysis, a frequently cited model evaluation leaderboard. The scores come from a composite benchmark and are not the same as the experience score users get in real workflows. fawn's original post is here: https://x.com/fawnunwrap/status/2113693298800074801 . According to that chart, Claude Opus 5.5 is 58, Claude Sonnet 5.5 and Muse Wonder Full are 56, Kimi K3.1 Pro is 55, Claude Fable 5.1 and GPT-6 Astra are 53, Gemini Argon high and Muse Wonder max are 52, and Grok 4.7 Sol is 46. Based on this, fawn said OpenAI is no longer in the top five. Some replies found that hard to accept, some directly questioned whether benchmarks can determine which model is best, and others said Muse is benchmaxxed and Kimi may be too.

Kimi K3.1 Pro's release information is more specific. Moonshot calls it its strongest 3T-class open-weight model, with a 1M token context, multimodality, a MoE architecture, 2.8T total parameters, and 104B active parameters. Official claims include the ability to write complete projects, make Unreal, Unity, and Godot games, and do parametric 3D modeling in Blender and CAD; hallucinations in financial scenarios are down 65%. It is already available on Kimi.ai, Kimi Work, Kimi Code, and the Kimi API. The Highspeed version is 2 to 3 times faster, and open weights are scheduled for November 4. AGTP's writeup also mentions that Claude Opus 5.5 was released on September 22, at $4 per million input tokens and $20 per million output tokens, and Anthropic says it is 40% cheaper than Opus 5 under typical workloads; on OpenRouter, Opus 5.5 has a median speed of 82 token/s, while GPT-6.1 Sol is 38 token/s. Some users also complain that weekly limits run out quickly. These numbers come from releases and writeups, and they are on a separate track from AA scores.

Nathan Lambert's reminder lands here: leaderboards can show potential, but they cannot replace trial use. A two-point gap among top models may reflect different task distributions, or differences in long context, tool calling, code-editing reliability, and cost and speed. One reply used an analogy: a benchmark is a spec sheet, while usability is a test drive. For developers, AA is good for screening candidates, but real tasks determine what they keep using; for ordinary users, response speed, limits, and price directly affect the experience. Running K3.1 Pro through the tasks you commonly do is more useful than staring at a bar chart.