Seven Trade-offs in MoE Routing: When Tokens All Crowd Toward the Hottest Experts, Latency Actually Gets Worse

In an MoE layer, the router scores each token and only lets the highest-scoring experts execute. Intuitively, the more concentrated the experts, the more compute you save. In practice, it can be slower: many tokens crowd toward the same set of experts while other experts sit idle; under fixed capacity, some assignments may be dropped; across GPUs, uneven work and communication overhead can eat up the compute savings. In this tweet and long-form article, Avi Chawla walks through seven production routing strategies, dispatch/combine, expert parallelism, and benchmark details. One commenter compared the problem to GPU rush hour; another reply was shorter: sparse model, dense traffic jam.

First, some background. In a dense transformer, every token passes through the same FFN. An MoE layer replaces this with multiple experts plus a router, and each token goes only to the selected experts. The Qwen3-30B-A3B model card states it clearly: 30.5B total parameters, 3.3B activated parameters, 48 layers, and each token selects 8 out of 128 routed experts. The 3.3B figure describes a rough compute estimate for a single token's execution path; it does not determine deployment memory. The serving side still needs to hold attention, embeddings, all expert weights, temporary buffers, and the KV cache. At 2 bytes per parameter for weights, 30.5B is about 61GB, or 56.8GiB, and that only counts weights.

Seven Routing Strategies

Token-choice routing: Each token selects its top-k experts based on router scores. The number of experts per token is fixed, while expert load is variable, so load-balancing mechanisms are needed to prevent traffic from piling onto a few experts.

Expert-choice routing: Each expert picks the highest-scoring tokens within a fixed capacity. Expert workloads are more balanced, but the amount of expert compute a single token receives may vary.

Dynamic routing: Different tokens select different numbers of experts. Confidence-based variants assign more experts when routing scores are ambiguous, but router confidence does not directly equal token difficulty.

Balanced assignment: Performs global matching to assign tokens under expert-load constraints. Some tokens may not reach their highest-scoring expert in exchange for a more balanced assignment, and the matching computation itself has overhead.

Hash-based routing: Maps tokens to experts using a deterministic hash, with no learned gating. This removes router learning, but expert selection is also less able to adapt to context.

Soft MoE routing: Experts process weighted mixtures of token representations through slots. A slot can merge information from multiple tokens, so what each expert processes changes.

Group-limited routing: The router first restricts the candidate expert groups, then selects experts. When group partitioning aligns with hardware placement, this can reduce cross-device or cross-node communication.

These choices affect how the serving engine batches expert computation and splits work across GPUs. Avi specifically warns that evenly distributing tokens does not guarantee low latency. Expert batches can still be very small, and moving tokens between devices takes time.

How a Batch Becomes Expert Matrices

Routing first produces an assignment for each token. If one token selects two experts, it produces two assignments. The runtime groups by expert and packs the token vectors, target experts, and routing weights. The resulting grouped matrices have different numbers of rows. Grouped GEMM runs multiple expert matrices in a single kernel launch, reducing launch overhead. Some engines pad expert matrices to a fixed size to simplify scheduling, at the cost of also doing computation on empty rows. Small decode batches expose the problem most clearly: when each expert has only one or two rows, kernel launch and data movement can cost more than the matrix multiplication itself. One commenter in the replies asked about wasted padding, and another said engineering practice uses dynamic grouping or sparse computation to mitigate it; Avi's long-form article does not go into detail.

Measurement needs to be broken apart: time routing, permutation, expert GEMM, and combine separately. A single overall MoE timer will not reveal where the slowdown is.

Expert Parallelism Keeps Weights in Place

Expert parallelism places different experts on different GPUs. If a token's assignment falls to a remote expert, its activations and metadata are sent over, computed, and sent back. Repeated across layers, all-to-all communication accumulates. DeepEP provides a throughput path and a low-latency path, aimed at large transfers and small decode steps, respectively. Network topology determines the cost across NVLink, InfiniBand, and Ethernet. Placing hot experts near the token source can help, but do not pile hot experts onto one node; once overloaded, the slowest GPU determines layer latency.

vLLM's EP deployment guide gives a concrete example: with TP=2 and DP=4, attention is split into 4 groups of 2 ranks each, and the expert layers can use an EP group of 8 ranks. Replicating experts can split traffic, but it consumes GPU memory and squeezes the KV cache. Rebalancing changes the physical destination without changing the logical expert selected by the router.

Tail Latency and the Limits of top-k

If one expert receives half the assignments, the GPU it lives on has to compute a larger matrix. Other GPUs that finish earlier still have to wait. Average utilization does not reveal this tail latency; you need to record, per serving step, the number of assignments, dispatch time, expert execution time, combine time, and GPU idle time.

Optimizations fall into two categories. Fusion, grouped GEMM, placement, and communication overlap only change how execution is performed. Quantization changes the numeric values used to store weights or transmit activations. Reducing top-k changes the set of executing experts, so outputs change. Avi cites the paper Training-Free Halving of Activated Experts: on Qwen3.6-35B-A3B, reducing from 8 experts to 4 loses 4.65 MMLU points with the standard approach, and 0.35 points after adjusting for reference quality; on Qwen3.5-397B-A17B, reducing from 10 to 5 loses 0.55 points. The paper covers only specific checkpoints and evaluations and should not be treated as a general rule. Capacity limits and token dropping also depend on the specific runtime; MoE inference does not automatically drop tokens just because experts are busy.

Troubleshooting Order

First confirm weight and KV cache capacity, then profile each stage, then measure assignment skew and bytes counted by communication domain, and finally compare placement, kernels, quantization, or top-k using the same request trace. Change only one variable at a time. Test both low and high concurrency, and both short and long prompts. Record TTFT, TPOT, p95 latency, and GPU-level idle time.

At the end of the long-form article, Avi asks: would your deployment check memory, expert compute, or communication first? There is no universal answer. Start by separating three things: storing weights, computing selected experts, and moving routed activations. Measure clearly which one is most expensive, then decide what to change.

Original: 7 MoE routing strategies, MoE inference engineering.