How to Fit a 3-Trillion-Parameter Model into 59 Wafers: Cerebras Lays Out the Memory Math

Cerebras released a video in which CTO Sean Lie and developer experience lead Sarah break down a question in the hardware lab: how does a 3-trillion-parameter Kimi K3 run on a wafer-scale engine? The conclusion first: 59 wafers. Each wafer has 44GB of on-chip SRAM.

Let's break the model down. Kimi K3 has 93 layers, of which 69 are linear layers and 24 are dense layers. The demo assumes a context length of 256K, weights compressed to 4-bit, and an 8-bit KV cache. In the video, Sean notes that these numbers are example estimates; actual production configurations will vary.

Weight accounting isn't complicated. 3 trillion parameters at 4 bits each, with 8 bits per byte, gives 1.5TB of weights total. Divided by 93 layers, that's about 16GB per layer. Each wafer has 44GB. Linear layers do not have a conventional KV cache, and their extra state can be ignored, so one wafer can hold 2 linear layers, using 32GB and leaving 12GB. Dense layers use global dense attention, and the KV cache is all here, so one wafer holds only 1 layer.

Where do the 59 wafers come from: 69 linear layers at two per wafer, 35 wafers; 24 dense layers at one per wafer, 24 wafers; 59 wafers total. In the video, black squares represent wafers, stacked layer by layer. This visual is more intuitive than a pile of peak TFLOPS: for deploying large models, you first look at memory capacity and how layers are partitioned, then at compute.

The comments didn't stop at applause. Some were surprised it was only 59 wafers and asked about cost. One commenter, using earlier cost assumptions, estimated that 20 CS4 servers, i.e. 59 wafers, would cost more than $100 million; an equivalent amount of high-speed HBM memory would be about $3.5 million. The estimate is rough, but the question is real: wafer-scale SRAM is expensive, and what it saves is memory bandwidth and power.

Others pointed out that quantizing the model from F16 to F4 reduces accuracy in exchange for inference speed. This trade-off is not new, but at 3 trillion parameters and 256K context, the accounting gets magnified. Tommy noted in a reply that on-chip KV still limits concurrent decoding. Daivd Yuan asked whether RDMA bandwidth would become a bottleneck. jalotra asked whether prefill is done on CPUs outside the wafer-scale engine, and then tokens are pushed to the first layer. T Assistant asked whether the remaining 12GB on the linear-layer wafers is wasted. These questions are more useful than 'crushing Nvidia.'

My take: this Cerebras breakdown is worth watching. The key is that it reduces deploying large models to three accounts: capacity, bandwidth, and concurrency. It is common knowledge that SRAM is small; wafer-scale makes per-chip SRAM 44GB, changing one premise of that common knowledge. The cost is also clear: models must be quantized, layers must be sliced, wafers must be stacked in number, and cost does not disappear. For developers, 59 wafers is a checkable anchor point; for ordinary users, a 3-trillion-parameter model landing in reality means 93 layers, 1.5TB of weights, 59 wafers, and a pile of unresolved engineering problems.

Original thread: https://x.com/seanlie/status/2115859011346342107