H100 Can Run 270,000 Threads at Once, but a Single if/else Makes It Queue

Gergely Orosz tweeted over the weekend that learning the GPU hardware layer is very educational. He shared the print edition of Modal's GPU Glossary, which @can gave him; the PDF is free. The photo of the book page discusses the differences between NVIDIA GPU streaming multiprocessors (SMs) and CPU cores, and also gives specs for the AMD EPYC 9965 and H100 SXM.

A commenter summarized the key numbers very clearly: the H100 has 132 SMs, each SM can hold at most 2048 resident threads, so about 270,000 threads are in flight at the same time. A CPU usually runs only a few dozen threads. This comparison explains many design choices. CPUs use large caches and branch prediction to hide memory latency; GPUs rely on many warps switching when they stall. A warp is 32 threads executing the same instruction. When an if/else branch occurs, the two paths must be taken one after the other, and CPU programmers easily get tripped up here. One commenter said, 'warp divergence tends to surprise CPU programmers,' because when threads in a warp take different branches, the two paths can only be executed serially.

The memory hierarchy is even more worth looking at. One commenter pointed out that the cost of moving data is far higher than that of mathematical operations; slow GPU code and strange inference pricing are often memory-bandwidth problems. Small-batch decode mainly waits on memory, not FLOPs. Another commenter simulated a small AI chip and found that even with the largest grid, the model couldn't fill it; fewer than 1% of units were busy. There wasn't enough work, so parallelism was just for show.

Someone hand-drew notes on GPU architecture; the accompanying diagram is more intuitive than text.

Modal released the GPU Glossary for free. One commenter joked that a physical book about GPUs is probably the closest most people will get to touching a GPU. The free PDF is here. People writing CUDA and building inference services can fill in the hardware layer, and ordinary users can also understand why GPUs are expensive and hard to fully utilize. Don't expect to learn just by reading; one commenter said it's hard to learn only by reading, but it's enough as a map.