The MoE Training Handbook Built on 800,000 GPU Hours: First Fix the “Log Shows No Errors but Training Is 25× Slower” Problem

Hugging Face has released an MoE training handbook: The Ultimate MoE Training Handbook. Dongxi’s recommendation is brief: an emphatic recommendation, full stop. The handbook is free and open, with the entry point at huggingface.co/spaces/huggingface/moe-blogpost.

Thomas Wolf offered a blunt assessment: of today’s 20 best open-source models, 18 are Mixture-of-Experts; Dense is basically over. He also said that most of the know-how for training a good MoE still remains in the heads of a small number of teams. So they spent 800,000 GPU hours running experiments and wrote down the process.

There is a public discrepancy in the figures; just be aware of it before reading: Thomas Wolf’s tweet says about 70,000 words and 55 interactive figures; Nouamane Tazi’s tweet says 30,000 words and more than 50 interactive figures, and gives the GPU hours as 1/2M. The numbers don’t line up, but they point to the same thing: the content is organized around the problems training teams will actually run into, the code is runnable, and the numbers come from their own runs.

MoE can be understood simply as: the model has many experts, and each token activates only a subset of them. The total parameter count looks large, but computation follows the activated part. The engineering pain is right there: how tokens are routed, how to handle imbalanced expert load, how to hide communication, and which parameters should use fp32. The handbook lists very concrete problems:

• How many experts to use, how wide each layer should be, and whether to use shared experts
• Half of a step is spent on all-to-all; how do you hide it?
• The log shows no errors, yet training is 25× slower—why?
• How to choose among softmax, sigmoid, and sqrt(softplus) routing, and why the router stays in fp32
• aux loss or DeepSeek’s loss-free bias
• Continue training from a dense checkpoint, or train from scratch?
• At inference time, can you use fewer experts per token?

Thomas Wolf added a conclusion about expert count in a reply: with fixed total parameters and activated parameters, finer-grained experts help, but there is an upper limit. In one ablation, loss kept dropping up to 128 experts, each with width 1/4, top-8, and then stopped improving. Released models mostly fall between 128 and 384 experts and top-4 to top-8. Beyond that, small GEMMs and more all-to-all eat into step time, so the optimal number on a cluster is usually less sparse than what the loss curve alone suggests. Details and Megatron flags are in this section.

The tweets include demo videos:

Nouamane Tazi’s tweet also has a video:

Leandro von Werra put it more directly: if you want to get into large-scale MoE training, this may be the most comprehensive resource available.

This handbook turns the gap in MoE training between “it runs” and “it runs correctly and fast” into searchable entries: communication, GEMM shapes, routing numerics, load balancing, checkpoint reuse, and inference configuration. For people doing training, these are a troubleshooting checklist; for general readers, it explains why MoE is increasingly common on open-source model leaderboards while the truly hard part lies in system details.

70k words—budget a few days. Don’t open it on Monday morning. It may be more practical to jump around based on the problem you’re stuck on first.