Wink Pings

Qwen4 Max Scores 43, 3 Points Below Kimi but 11% Cheaper—Is That a Good Launch?

The AA Index gives Qwen4 Max 43 points, 9 points above Qwen3.8 Max and 3 points below Kimi K3.1 Pro, at $3.27 per task. Chubby says this is not a good release; the online debate is more worth watching than the scores.

After Qwen4 Max launched, Chubby♨️ wrote "not a good release" in a tweet. The controversy is not about whether there is progress, but about the mismatch between the story told by paper benchmarks and the one told by the AA Index. Artificial Analysis's AA Intelligence Index gives Qwen4 Max 43 points. Qwen3.8 Max xhigh is 34, and Kimi K3.1 Pro max is 46. Qwen4 Max is 9 points above the previous generation and 3 points below Kimi. Looking only at the score, it is in the upper-middle range—far from a complete failure, and also far from topping the chart.

Chubby推文截图

In AA's notes, Qwen4 Max uses xhigh reasoning effort and improves over Qwen3.8 Max on all 9 evaluations: HLE-Diamond from 24% to 45%, AA-Omniscience +18, AutomationBench-AA +17, CritPt v2 +13, Terminal-Bench-Science +11, Terminal-Bench 4.0 +9, GDP.pdf +3, FrontierCode 2.0 +3, and AA-Briefcase +82 Elo. That is not a small improvement. The problem is the horizontal comparison: Kimi K3.1 Pro is still ahead.

AA指数汇总图

Cost is the more interesting part of Qwen4 Max. AA calculates per-task cost at $3.27. Qwen3.8 Max is $10.02, and Kimi K3.1 Pro is $3.68. Qwen4 Max is about 11% cheaper than Kimi and about one-third the cost of the previous generation. On price, Qwen4 Max is $1.50 per million input tokens and $4.50 output, with cache hits at $0.075. Qwen3.8 Max is 2.00/6.00, with cache hits at 0.25. AA says the cost drop comes from both lower prices and lower token usage: Qwen4 Max outputs about 105k tokens per task versus about 171k for Qwen3.8 Max, 39% less; input tokens for running the Index are 9.9B versus 14.5B. Agentic evaluations resend context each round, and input accounts for 85% to 90% of cost, so a 70% drop in cache price has a direct impact.

成本与token图

The capability breakdown better shows what type of model this is. AA-Briefcase evaluates agent work in real business documents; Qwen4 Max gets 1686 Elo, Kimi K3.1 Pro 1674, and Tencent Hy4 1446. Qwen4 Max is not weak at knowledge work. Terminal and coding agents are the other side: on Terminal-Bench 4.0, Qwen4 Max is 48%, Kimi K3.1 Pro 67%, and Tencent Hy4 37%. On FrontierCode 2.0, Qwen4 Max is 26%, tied with Kimi and behind Tencent Hy4's 31%. As a general knowledge-work model, both the data and the price make sense; as a mainstay for terminal coding, the weakness is obvious.

评测细分图

Chubby's criticism focuses on distillation. He says Qwen has had its distillation channels cut off, which may explain why this release fell short of expectations. That claim is open to debate. Some users don't fully agree, arguing that distillation hasn't been completely cut off—only the target has changed—and that some open-weight models have an OpenAI flavor. Others aim their criticism at AA, saying the benchmark is biased and that they can't reproduce the same results. Still others do the math: 3 fewer points, 41 cents cheaper per task, acceptable for back-office scenarios. Justin Draplin's analogy is more direct: benchmarks are dating profiles, and the AA Index is the first date. The first date isn't perfect, but the bill is indeed cheaper.

网友讨论截图

There is another easily overlooked limitation: Qwen4 Max is not open-weight. AA's model page lists it as "proprietary", with weights unavailable. For API users, the roughly 1M context and text and image input are selling points; AA's page says 984k. For the open-source community, there's no new toy this time. Some are looking forward to a later 27B version; that might be the more exciting release.

Qwen4 Max looks more like a cost-optimization and knowledge-work-oriented release. AA Index 43, 3 points below Kimi, 11% cheaper, and still ahead on AA-Briefcase. Chubby says it's not a good release, and the part of that statement that holds is about expectation management: the gap between paper benchmarks and the AA Index, plus rumors of restricted distillation, makes the launch narrative look bad. Saying it's bad enough to be unusable is not supported by AA's data. People with a development background can look at token efficiency and cache pricing; ordinary users can look at the API bill. Leaderboards are the first date: Qwen4 Max's date performance is average, but when it comes time to pay the bill, it is quite restrained.

Reference links: original tweet, AA tweet, AA model page

发布时间: 2026-11-17 01:29