Token unit price fell 67%, but bills rose 120%: Where did the savings go?

a16z's chart this week puts two numbers together: since May, the average token price has fallen 67%, while token spending has risen more than 120%. Unit price and total bill are moving in opposite directions.

The subtitle of Figure 1 is direct: how much token spending has grown depends on whom you ask. The blue line is spending inferred by Ornn/OpenRouter, the red line is Ramp total spending, and the green dashed line is blended price. By October, the blue line had cumulatively risen 159.5%, the red line 122.1%, and the price had fallen 67.1%. The a16z tweet used the lower +120% measure. Some commenters in the replies pointed out that the two spending lines differ by 37 points; based on the Ornn line, usage since May is close to 8x. Different definitions can sway the conclusion, but the direction is consistent: the price fell, but spending did not.

The second chart is more interesting. From mid-July to October 26, B300 rent rose 106%, B200 rose 65.7%, and H100 rose 16.9%, while blended token cost fell 58.4%.

a16z calls this "Rack Spread": token prices go down, GPU rents go up. Some commenters calculated that B300 rent is about 2.06x, while token price is 0.42x, a gap of nearly 5x. If throughput has not increased by the same proportion, the profit margin of those renting GPUs to sell tokens is squeezed. Others asked whether large-capacity deals have long been locked into multi-year fixed-price contracts, and how much of the spot GPU price increase actually enters lab costs. This question is harder to answer than the surface numbers suggest.

Cheap tokens have not made bills cheaper, and the reason is not mysterious. Lower unit prices change usage behavior. One commenter said that after tokens became cheap, they stopped trimming prompts and started running five agents in parallel. Another said that an agent will first reread the entire repo, then change one variable name. Price fell 67%, usage rose 6 to 7 times, and the bill naturally goes up. The Jevons paradox was repeatedly mentioned in the replies: as resource efficiency improves, total consumption actually increases.

Measurement definitions are also creating illusions. One commenter pointed out that the a16z article itself says the sample is biased toward people who route to the cheapest models, and part of the 67% price decline comes from changes in the model mix. In other words, a lower average unit price does not exactly equal the same model becoming cheaper. The 37-point gap between the two spending lines in Figure 1 also shows that how much spending has grown depends on sampling and definitions.

For developers, this comes down to two metrics: price per token, and cost per deliverable. The former is falling; the latter may not be. If agents run more rounds, read more files, and try more approaches, cheap unit prices will be eaten up by the number of calls. For ordinary users, subscription fees will not automatically fall just because token prices fall; service providers need to cover GPUs, electricity, and fixed capacity.

One commenter said: what is cheap is each token; what is expensive is the habit. This sentence is closer to the charts than "AI costs are falling." What is worth watching next is whether, as token unit prices continue to fall, the spending curve will bend with them, or whether usage will keep pushing it up.

a16z original article and charts: Charts of the Week