MoE Computes Fewer FLOPs but Latency Goes Up: The Slowdown Is in Moving Tokens

An interview question: replace the dense feed-forward layer in a Transformer with a top-2 MoE. The profiler says fewer FLOPs are executed per token, yet end-to-end inference latency rises. Why?

Avi Chawla broke down this path in an article on MoE inference engineering. First, separate two concepts.

The difference between dense and MoE

In a standard Transformer layer, attention first mixes information across token positions, and then the FFN independently transforms each token. All tokens go through the same FFN with the same weights. After replacing the FFN with MoE, a layer has multiple experts, each an independent FFN. The router reads each token's hidden state, scores the available experts, selects the top-2, and produces two routing weights. Only the selected experts process that token. The model stores many expert weights, and each token executes only a small fraction of them.

So the profiler seeing per-token FLOPs drop makes sense. The problem is end-to-end. MoE saves expert computation, but it does not save movement and scheduling.

There is a chain of operations after the router

top-2 routing means one token produces two assignments. A batch with N tokens produces 2N assignments. At runtime, they are grouped by expert: which tokens E1 receives, which tokens E2 receives, forming matrices of different sizes. This grouping is called dispatch. Sorting token rows by expert is called permutation. Each expert runs its own matrix, usually with grouped GEMM combining multiple expert matrices into one kernel launch. The results are then restored to token order with inverse mapping, multiplied by routing weights, and added; this is called combine and unpermutation.

In prefill, many tokens arrive at once, expert matrices have many rows, and the GPU is well utilized. In decode, each request adds only one token at a time; when concurrency is low, an expert may receive only one or two rows. The costs of kernel launch, weight reads, and moving token rows emerge. grouped GEMM reduces launches, but it cannot create work for experts with no assignments. Fixed-shape padding also counts empty rows. A profile that only looks at FLOPs will miss these segments.

When crossing GPUs, activations have to go out

When there are too many experts to fit on a single card, expert parallelism is needed. Different experts are placed on different GPUs. A token on GPU 0 selects an expert on GPU 1, so the activation vector and routing metadata must go over the network, be computed, and come back. In a batch, different tokens select different experts; each GPU may send activations to multiple GPUs at the same time and receive activations from multiple GPUs. This pattern is called all-to-all.

The video in the source material traces the path for one batch. T1 selects local E1, so the activation stays on GPU 0. T2 selects E4 on another GPU in the same server, so it goes over NVLink. T3 selects E7 on another server, so it goes over InfiniBand or Ethernet. Every MoE layer repeats this exchange. Cross-server links are more expensive than intra-machine links, so latency accumulates.

The slowest GPU determines this step

Routing is unlikely to partition work perfectly evenly. A hot expert receives half a batch's assignments, so the GPU hosting it has to compute a larger matrix. Other GPUs may finish early but still have to wait. The step latency of a MoE layer is determined by the slowest participating GPU. Average utilization hides the tail. Record assignment counts, dispatch time, expert execution time, combine time, and GPU idle time per serving step, and also record bytes and time per communication domain. Mixing intra-machine NVLink and cross-machine InfiniBand into one total counter makes it impossible to see where the problem is.

placement can put experts that are often selected together closer to each other, and put hot experts on less busy GPUs. replication can create multiple physical copies of one logical expert; router choices stay the same, and at runtime assignments are sent to available copies. Both approaches consume GPU memory and control bandwidth. Traffic spikes may end before rebalancing completes.

Memory is calculated by total parameters

MoE has another accounting detail that is easy to confuse. The Qwen3-30B-A3B model card lists 30.5B total parameters and 3.3B activated parameters, with each token selecting 8 of 128 routed experts across 48 layers. The 3.3B describes the execution path for one token; deployment must store 30.5B total parameters, attention, embedding, temporary buffers, and KV cache. At 2 bytes per parameter for weights, 30.5B requires about 61 GB, not counting KV cache and communication buffers. Sparse activation reduces expert computation, not weight storage. Activated parameters and resident parameters must be measured separately.

Optimization starts with measurement

There are several categories of optimization. Put frequently selected experts in the same NVLink domain to reduce remote traffic. Make routing more balanced so one GPU does not hold up the whole layer. Overlap remote activation transfer with local expert computation. Use grouped GEMM to reduce kernel launches. Quantize weights or activations to reduce memory and network bytes, but measure numerical error. Changing top-k changes the set of experts executed and is a model behavior change; compare quality on the same batch of requests. Placement and kernel changes alter the runtime, quantization changes numeric values, and top-k changes the computation set; these types of experiments should be kept separate.

End-to-end latency is not settled by FLOPs. It is settled by the slowest link. MoE makes computation sparse, and it also pushes tokens onto the road. The FLOPs missing from the profiler may turn into dispatch, all-to-all, and waiting.

Reference: Avi Chawla's MoE inference engineering article https://x.com/i/article/2100852768403861504 , original tweet https://x.com/DailyDoseOfDS_/status/2118999411254767695 .