Same Score, 10x the Bill: Agent Arena Lays Out Cost, Tokens, and Wait Time

Arena.ai updated Agent Arena on X. It now not only tells you how much a task cost, but also shows performance per output token and per minute waited.

The question in the post is straightforward: at the same intelligence level, cost can be 10x higher, output tokens 6x higher, and time taken 2x longer. Agent Arena uses a Pareto frontier to plot net improvement against resource consumption.

The chart's x-axis is resource amount, with the first point on the frontier set to 1x and a logarithmic scale extending to 1000x; the y-axis is net improvement percentage, from -5% to 20%. It aims to answer how much improvement is gained in return for extra money spent, extra tokens generated, and extra time waited.

The first thing one can read from the chart is that cost rises most steeply with intelligence level. Waiting longer does not yield uniform gains. The number of output tokens has very weak predictive power for the bill.

A commenter, Dusan, wrote in a reply: same result, 10x the cost. Another reply came from Jatin Garg, who said that when pricing by successful tasks, retries should be counted; otherwise, cheap but failed attempts will look like efficient agents.

This comment hits on a pricing-methodology issue. If cheap but failed attempts do not count retries, they will be seen as efficient agents. If the leaderboard only looks at single calls, readers can easily miss the cost of failure.

David Batista broke down the numbers more carefully: output tokens are 6x, the price per token is about 1.7x, and multiplied together they come close to 10x the task cost. His calculation rests on the assumption that output tokens drive the bill.

He also raised an open question: does the time view count tool-call waiting, or only generation time? This definition affects judgments about agent latency.

More worth looking at is that the leaderboard lays out cost, tokens, and time separately. For people building agents, looking only at scores when choosing a model will miss the bill structure and waiting costs. For ordinary users, this set of charts also explains one thing: answers that look similar in the chat box may consume completely different resources behind the scenes.

Arena.ai provided a Pareto leaderboard link. If you need to choose among several agents, first look at whether your bottleneck is money, tokens, or time, and then look at the corresponding curve.