GPU compute keeps piling up—why is token generation still slow

AI hardware has more than one answer today. Daily Dose of Data Science laid out five architectures side by side: CPU, GPU, TPU, NPU, and LPU. The point is not which one replaces which; every path makes trade-offs among flexibility, parallelism, and memory access. The video below is the architecture comparison from the original post:

CPU takes the general-purpose route. A few powerful cores, deep caches, and strength in complex logic, branching, and system tasks. It is well suited to databases, operating systems, and decision-heavy code. Repeated math such as matrix multiplication is not its strong suit.

GPU is the opposite. Thousands of smaller cores execute the same instruction on different data. The large amount of matrix math in neural network training and inference maps neatly onto this kind of parallelism. The cost is weak single-thread performance, and data-dependent branches make threads in the same warp diverge, forcing the hardware to run two paths serially.

TPU goes further toward specialization. At its core is a MAC array; weights and activations flow in from both sides, and partial results propagate through the array without having to return to memory every time. Execution is controlled by the compiler, not by hardware scheduling. Google designed this structure for neural network workloads.

NPU is the edge version. A neural compute engine plus MAC arrays and on-chip SRAM, using low-power system memory, aims to run inference within a single-digit-watt budget. Phones, wearables, and IoT are its place. Apple Neural Engine and Intel's NPU belong to this direction.

LPU comes from Groq and is a more recent path. It removes off-chip memory from the critical path and places all weights in on-chip SRAM. Execution is fully deterministic, scheduled by the compiler, with no cache misses and no runtime scheduling overhead. The cost is limited memory per chip; serving a large model requires hundreds of chips connected together. The latency advantage is real.

From CPU to LPU, generality is traded step by step for efficiency. This trade-off is more worth looking at than which one is stronger.

But in real deployments, the more common confusion happens on GPUs: spec sheets list close to a quadrillion floating-point operations per second, yet running a 70B model to generate text yields only a few dozen tokens/s. In a long article, Akshay explains this phenomenon with an asymmetry: arithmetic is cheap, fetching data is expensive. Arithmetic units are like workbenches in a workshop; the data is in the warehouse, and between them is only a narrow corridor. No matter how many workbenches you add, if the corridor width stays the same, throughput is stuck at the corridor. That corridor is memory bandwidth.

GPU does not shorten the wait; it lets the wait be covered by other work. A large number of warps reside in each compute unit; when one warp waits for data, the scheduler switches to another ready warp. Arithmetic units stay busy. But bandwidth is a hard ceiling: only so many bytes can pass per second, and switching cannot hide that.

So to tell whether an operation can saturate a GPU, calculate work per byte, that is, arithmetic intensity: how many operations are performed, and how many bytes are fetched from main memory. H100 SXM5's public specs are 989 TFLOPS dense BF16 and 3.35 TB/s bandwidth; dividing the two gives about 295 ops/byte. Below this ratio, no matter how high compute is, it will wait on memory. Above it, you enter compute-bound territory.

LLM token generation lands exactly on the worst side. A 70B model at 16-bit precision is about 140GB of weights. For every token generated, the weights must be read in full once, and each weight does one multiply-add, about two operations. 140 billion operations divided by 140GB of traffic is about 1 op/byte, three hundred times lower than 295. So token rate is basically determined by bandwidth: 140GB divided by 3.35TB/s is about 42ms, or about 24 tokens per second. This lower bound does not disappear by changing code.

The prefill phase is the opposite. Multiple tokens in the prompt are processed together, the same weights are reused by multiple tokens, work per byte rises, and it usually becomes compute-bound. Long prompts and long outputs have different bottlenecks; lumping them together easily causes confusion.

Most optimizations that have appeared only move two variables: increase the amount of computation per fetch, or reduce the bytes fetched. Batching increases computation. Ten concurrent requests share one weight read, multiplying work per byte by ten. At 16-bit precision, it takes roughly 300 concurrent sequences to push generation past the compute-bound line. Quantization reduces bytes. 8-bit weights shrink a 70B model to about 70GB, and the theoretical single-sequence upper limit rises from about 24 tokens/s to about 48 tokens/s, at the cost of precision. Fusion and FlashAttention reduce round trips of intermediate results to main memory. None of these methods is magic; they just make the same accounting more favorable.

There is another misleading metric: GPU utilization in monitoring. It often only shows that work has been scheduled, not that arithmetic units are doing useful work. A GPU that is 100% utilized but waiting on memory and a GPU at full throughput may show the same reading. Look at batch size, precision, and KV cache size first; that is usually more useful than staring at FLOPS.

Hardware will keep changing. H200 and H100 use the same compute die; compute remains 989 TFLOPS, bandwidth rises to 4.8 TB/s, and the break-even ratio drops to about 206 ops/byte. More workloads will become compute-bound. But the shape does not change: arithmetic grows faster than bandwidth, technologies that reduce data movement will become more valuable, and peak compute on spec sheets is often the number that least predicts actual performance.

Punit Vara explains CUDA kernels in a Hinglish rap, telling the story of threads, blocks, and GPU parallelism. The video is here: https://youtu.be/i_ZBgKXLrtM

When looking at a chip architecture diagram, first ask what it puts on-chip and what it pushes off-chip. The answer is often closer to real speed than TOPS or TFLOPS.