Training costs 1 FLOP, inference costs 6 a year: YC walks a token from API to GPU

YC's Paper Club this week is about inference. First, two numbers: model training costs 1 FLOP, then serving it each year costs about 6 FLOPs; over the past decade, getting this inference layer right has generated cumulative market value of more than $220 billion for related companies. A commenter summarized that training usually gets the launch tweet, while inference gets the 3 a.m. alert. That's pretty accurate.

Inference doesn't end with one model.forward call. A token enters through an API request and must pass through routing, caching, scheduling, kernels, VRAM, and hardware, with trade-offs at every step among speed, cost, and quality. YC brought together researchers and engineering leads from Together AI, Ollama, Luminal, SGLang & RadixArk to explain it along the end-to-end flow. Video here:

This session is broken down by inference stack: 00:00 Why inference matters more than training; 08:24 Three types of inference innovation; 12:14 Dan Fu on the lifecycle of an inference request; 42:18 Jeff Morgan on why not all tokens from open models are the same; 1:04:31 Joe Fioti on compiling models with search instead of heuristics; 1:30:30 Liangsheng Yin on SGLang and memory in the agent era. Full YouTube version is 1 hour 50 minutes, link here.

A few specific points are worth noting. The same open model can perform differently with a different provider. A commenter asked whether it's quantization or low-level kernels; that question is exactly the topic of Jeff Morgan's section. Caching in the inference stack is not just about speed. Agent sessions are a tricky case: the cache covers all previous tokens, and when a session is idle it still occupies memory it isn't using. The longer a chat goes, the more painful the KV cache becomes. A commenter said they'd like a dedicated section on KV cache. There's also energy: one commenter mentioned the 6:1 ratio matches their own workload, but electricity prices determine whether unit economics can break even. This angle came up repeatedly in the discussion.

Someone added an East African perspective. A commenter wrote from Kampala that for local small companies, AI is first of all an inference bill paid in U.S. dollars. Africa accounts for less than 2% of global data center capacity, about 1.2 GW of IT load in 2025 and projected to approach 3.5 GW by 2030; about 600 million people still lack electricity. The good news is costs are falling. Stanford HAI 2025 AI Index data shows that from November 2022 to October 2024, the cost of running GPT-3.5-level systems fell by more than 280x. So that person's advice is practical: route by task—send simple summaries, tagging, and drafts to small models or local Ollama, and pay frontier-model prices only for hard reasoning; cache repeated Q&A; treat the AI budget as a usage-based cost line item rather than a one-time purchase. Original text is in this AVODA analysis.

This 6:1 ratio is a number YC gave; some replies asked what data it's based on, and it's worth checking later whether the video expands on it.

YC also posted the next Paper Club, on November 4, about AI for biology, link here.

Inference is discussed less because it hides behind APIs, and users only see the price per million tokens. But the same model with different providers, the same session with different caching strategies, and the same GPU with different kernels all ultimately turn into bills and latency. Training determines what a model can do; inference determines whether it can be used affordably. This video breaks open the engineering accounting in that middle layer.