Four GPUs Do Not Equal 96GB: What LLM Inference Parallelism Is Actually Sharding
Four GPUs have four independent memory pools. Data, tensor, pipeline, context, expert, and hybrid parallelism respectively shard requests, weights, layers, sequence positions, and expert weights. Choose the wrong strategy, and communication overhead will eat the memory and compute you saved.
Four 24GB GPUs plugged into one machine do not become 96GB. Each card has its own memory and processor. If a kernel needs to read weights or a KV cache block from another card, the data must first cross the interconnect. LLM inference parallelism starts here.
LLM inference has two phases. prefill processes the prompt and builds the KV cache, affecting TTFT. decode generates token by token, repeatedly reading weights and the growing KV cache, affecting TPOT. prefill is more about large matrix computation, while decode is more about memory bandwidth. A layout that shortens prefill may add synchronization to every decode token.
The resources that usually limit deployment fall into four categories: parameter memory determines whether the model fits, KV cache memory determines resident tokens and concurrency, GPU compute time, and communication time. Metrics need to be viewed separately: TTFT, TPOT, request throughput, token throughput, goodput, and tail latency such as p95. In his article, Avi Chawla warns that average throughput hides tail latency; a saturated server can report high throughput while violating latency targets.
Parallel strategies are about placement. Each method shards different state and brings different communication. collective means a group of GPUs exchange or combine data together, for example all-reduce sums partial results from each card and then lets participating cards receive the complete result. A slow card or slow link will hold back the whole group.
Data parallelism DP replicates the full model, and routing distributes requests to replicas. It improves throughput, but it does not solve the problem of a single model not fitting. Check three things: whether a full replica already fits; whether routing only looks at round-robin and ignores queues, idle KV, and prefix hits; whether admission reserves KV for decode growth. At low traffic, the second replica sits idle.
Tensor parallelism TP shards the matrix operations inside each transformer layer. The Megatron-LM two-projection construction is a common example: the first weight is sharded by columns, the second by rows, each rank computes a partial output, and then all-reduce merges them. TP2 splits an 8192-column matrix into 4096 columns per GPU, and TP4 gives 2048 columns per GPU. Per-GPU weights and local computation shrink, but every layer needs a collective, and this also happens at every decode step. The higher the TP degree, the smaller the local matrices and the higher the communication share. The engineering rule is to use the smallest TP degree that satisfies memory and latency, and place TP on the fastest links. On a four-GPU server, try TP2 first, leaving the other two cards for a second TP2 replica; if TP2 still does not satisfy memory or single-request latency, then consider TP4.
Pipeline parallelism PP shards layer groups by depth. stage0 runs the first layers and sends hidden states to stage1, and the final stage outputs logits. Inference has no backpropagation, and continuous batching can fill stages, but a single sequence still depends on the previous token. PP communicates less frequently than TP and is suitable across slow links, but it does not automatically reduce single-request latency. Check whether there are enough independent requests to fill the stages, and check stage balance based on measured execution time and memory, not equal layer counts. Four-card PP4 is only reasonable when the model truly needs four cards and concurrency is sufficient; two pairs of fast cards can use PP2 plus TP2 within each segment.
Context parallelism CP shards sequence positions. PCP shards prompt tokens, with the goal of lowering TTFT. DCP shards KV history, with the goal of increasing cache capacity. In DCP, each decoder layer must merge partial attention and softmax statistics, using global maximum rescaling during the merge to avoid exponential overflow. The cost is that TPOT may rise. PCP is measured with long-prompt buckets for TTFT; DCP is measured with KV capacity and concurrency. Normal chat stays on the baseline, and long prompts are routed to the PCP group.
Expert parallelism EP is only meaningful for MoE models. The router selects experts for tokens, token representations are sent via all-to-all to the ranks that own the experts, and then returned after computation. It saves expert weight memory, but puts data-dependent all-to-all and load imbalance on the request path. Check per-rank expert weight memory, transfer bytes and all-to-all time under the real prompt distribution, and also look at max tokens per expert and wait time. Average throughput hides stragglers.
Hybrid parallelism combines the methods above. The order can be like this: first choose the smallest TP group so that each stage fits, and place it on the fastest links; add PP only if the model must cross a slow boundary; after one complete instance fits, add DP; add CP if long context has been measured; if MoE expert weights are a fit problem, try EP. One candidate for eight GPUs on two nodes is intra-node TP4 and cross-node PP2, keeping frequent collectives inside the node and activating the slow network only at stage boundaries. If the model fits on four GPUs, TP4 plus DP2 forms two replicas, and requests do not depend on the cross-node path.
The decision framework starts with the smallest change. First check whether a single replica can fit on one GPU while preserving the KV needed for target concurrency. If yes, start with one replica, add DP as traffic rises, and have routing preserve prefix locality. If not, find the smallest TP degree and benchmark decode latency at that degree. If layer-group execution times are close or peer links are slow, consider PP. When crossing nodes, keep TP inside the node and PP across nodes, and first measure stage balance. Treat long prompts separately for PCP and KV capacity separately for DCP. Treat MoE separately for EP, and track token imbalance and all-to-all separately. Add only one axis at a time, with each axis having an explicit task and measurement validation. Parallelism is a placement decision, not a GPU-count setting. Use the simplest layout that meets latency and throughput targets, and keep measuring the alternatives. Some netizens mention that PCIe allreduce on 4090 is the real wall; this is the same lesson as putting frequent collectives on the fastest links.
Avi Chawla original article: https://x.com/i/article/2101395766170533888
Tweet: https://x.com/DailyDoseOfDS_/status/2123083871986294929
Quoted tweet: https://x.com/_avichawla/status/2107427400779026843
发布时间: 2026-11-19 17:41