73 Points, Tied for First, but Bills Differ 7x: The Decision Model Leaderboard Starts Penalizing “Confidently Wrong” Answers

Artificial Analysis has released Decision Index v2. The top three decision models on the leaderboard differ by only 1 point, yet their bills differ by more than 7x. Drex Max v1.1 and GLiDE v1.2 both score 73, while Mammoth 27B scores 72. On cost, Drex Max v1.1 costs $102 per million tasks, while GLiDE v1.2 costs $780.

Decision Index measures a class of models built specifically for making judgments. They choose an answer from fixed options, assign a probability to each option, and complete the task in a single call. They are suited to tasks such as routing, triage, and document judgment. Language models can also do this, but they generate structured output, so their cost and latency curves are different. v2 combines document decisions, tool calling, knowledge, reasoning, and equivalent-question consistency into one weighted score. 45% of the evaluations are based on private datasets, compared with 25% in v1. This change is intended to reduce room for leaderboard gaming.

New Metric Rewards Calibration

v2 switches its main metric to Decision Score. Each question is scored from 0 to 100. Correct answers fall between 50 and 100, and incorrect answers fall between 50 and 0. The more confident the model is in its top answer, the farther the score moves from 50. In other words, being correct and confident adds points, while being wrong and confident is penalized more heavily. A user wrote in a reply that being confidently wrong is the most expensive mistake. This observation is very apt for decision models. If a routing decision is wrong, humans can still catch it; if the model is very confident and routes incorrectly, the cost is higher.

AA-Paperclip Accounts for 30%

The newly added AA-Paperclip has the highest weight, at 30%. It contains 1,500 private documents: résumés, receipts, invoices, and AI and machine learning research papers. Each document has 5 decisions and 10 fact extractions, for 15 answers total. Models must be scored jointly across all 15 answers as a single decision, and only all-correct counts as a pass. In Decision Score, document confidence is the product of the probabilities of the 15 answers.

Pass rate magnifies this gap. Claude Haiku 5.5 Max passes 92% of documents, and GPT-6 Luna Max passes 86%. Among decision models, Drex Max v1.1 passes 63%, GLiDE v1.2 passes 62%, and Decider v1.2 passes 55%. Drex Max v1.1 has 97% answer accuracy, close to Claude Haiku 5.5 Max’s 99%, but its document pass rate is only 63%, versus 92% for Claude. The gap is mainly lost on decisions. Drex Max v1.1 gets all 5 decisions right in 71% of documents and all 10 extractions right in 88% of documents. High per-item accuracy does not necessarily mean a high joint-decision pass rate. This design is closer to real document work.

Equivalent-Question Consistency

AA-Symmetry accounts for 15%. It uses 120 base questions to create 440 variants, with changes including reordering options, swapping labels, and turning yes/no questions into negated phrasing. Models must keep answers and high probabilities consistent across different phrasings. Base Labs’ models lead thanks to a high abstention rate. Mammoth 27B scores 96, and Elephant 9B scores 93. Mammoth 27B chooses the same answer on 99% of questions, Drex Max v1.1 is at 91%, GLiDE v1.2 is at 88%, and Clef is at 67%.

Here is a detail to watch when reading the leaderboard. High consistency does not equal good task performance. If a model often admits it doesn’t know and is very confident about not knowing, it can also score high on consistency. AA-Symmetry is valuable, but it measures stability. When looking at the overall leaderboard, you have to break out the abstention rate.

The weights were adjusted as well. AA-Omniscience-Decision v1.1 and HLE-MultiChoice v1.1 each account for 15%, down from 25%. AA-Omniscience-Decision replaces the “I don’t know” option with ten different phrasings. HLE-MultiChoice added refusal options with the same ten phrasings. EvalSafe-ONET-AA accounts for 15% and is Typesafe’s implementation of the ONET benchmark. NVIDIA’s When2Call is retained, with its weight reduced from 25% to 10%. WildGuardTest was removed due to saturation.

The Cost Trap Is in the Billing Structure

The cost per million tasks for decision models varies by more than 60x. Celeris-1.1 Decision is $14, Decider v1.2 is $68, and Drex Max v1.1 is $102; the three sit on the Pareto frontier of Decision Index versus cost. Clef is $859. GLiDE v1.2’s list price is only 33% higher than Drex Max v1.1, but its cost per million tasks is $780, nearly 8x. The reason is the billing structure. Most decision models bill a document’s input once and answer all questions. GLiDE v1.2 and Decider Flash v1.2 run each question separately and re-bill the input every time. For one AA-Paperclip task, GLiDE v1.2 uses about 63k input tokens, while Drex Max v1.1 uses about 9k. Decider Flash v1.2 therefore costs $196, nearly three times Decider v1.2, even though its token unit price is only half.

A user wrote in a reply that repeatedly charging the document fee per question deserves a warning label. Looking only at token unit price when selecting a model will lead you into a trap. Billing structure determines real cost more than unit price.

Language Models Score Higher—and So Do Their Bills

With language models added to the leaderboard, Claude Haiku 5.5 Max scores 77 and GPT-6 Luna Max scores 76, about 3 points higher than Drex Max v1.1. The tradeoff is cost and latency. GPT-6 Luna Max costs $1,530 per million tasks; Drex Max v1.1 is $102, 15x. GPT-6 Luna Max takes 11.34 seconds per task, while Drex Max v1.1 takes 275 milliseconds—about 40x slower. Claude Haiku 5.5 Max costs $6,240 per million tasks and takes 32.65 seconds per task. Celeris-1.1 Decision is the fastest at 56 ms, Elephant 9B is at 70 ms, and Jev 1.14 is at 92 ms. Mammoth 27B scores 4 points lower than GPT-6 Luna Max but is 118x faster.

These numbers show that decision models and language models solve different problems. Language models can score higher and are suited to complex reasoning and open-ended output. Decision models are cheap, fast, and have calibratable probabilities, making them suitable for high-frequency judgments. The leaderboard puts the two curves side by side; the choice depends on how much latency and cost the business can tolerate.

Where the Leaderboard Is Useful and Where to Be Careful

v2 incorporates confidence into the main metric, which is the right direction. Decision models’ biggest fear is being confidently wrong; calibration is closer to real-world value than accuracy alone. The 45% private data also makes leaderboard gaming harder. However, AA-Paperclip accounts for 30% and is entirely private; a user asked whether a held-out slice could be released for other labs to reproduce. With only 1 point separating the top, this question is not unreasonable. Another detail is AA-Symmetry’s reward for abstention rate; when reading the overall leaderboard, you have to break it out.

Full results are at Artificial Analysis Decision Index. The original announcement is on X.