Prime Intellect Adds Backpropagation to Fused MoE Kernel, Speeds Up Qwen3 Training Steps by Up to 34%

Prime Intellect has released Prime Mega MoE, a backpropagation implementation for DeepSeek's fused Mega MoE kernel. It makes the kernel trainable, and delivers up to 34% faster Qwen3 MoE training steps in the prime-rl framework. The original announcement also lists another key performance figure: a 3.11x speedup for the forward pass.

MoE models only activate a subset of experts per run, leading to far more fragmented computation paths and communication patterns than dense models. Fused kernels work by merging multiple small operators together, to cut down kernel launch overhead and redundant memory round trips. This optimization is already widely adopted on the inference side, but it was far more complex to implement for training, since the training pipeline requires both forward pass and backpropagation, and the backpropagation stage additionally needs to compute gradients and update parameters. A speedup limited only to the forward pass cannot support a complete closed-loop training workflow. The backpropagation support Prime Intellect added this time enables the optimization to truly integrate into full training pipelines.

The two performance metrics are measured under different conditions and should not be mixed up. The 3.11x speedup refers to the standalone forward pass, while the 34% speedup refers to real Qwen3 MoE training steps running in prime-rl. For developers, the latter metric is much closer to real end-to-end training performance gains. When evaluating this type of kernel optimization, standalone microbenchmarks often show inflated impressive speedup ratios, but actual training step latency and throughput are the metrics that reflect real production value.

Prime Mega MoE is a low-level engineering-focused release with no flashy public demos. It directly solves the two core problems for fused MoE kernels in training: whether the kernel is usable for training workloads, and how much performance gain it can deliver. This type of update is well worth following for teams building MoE training and inference stacks. For general readers, you can understand it this way: model training requires two full paths, the forward pass and the backpropagation, and this update has now fully optimized and opened the backpropagation path that was previously incomplete.

Source: MarkTechPost, Author Asif Razzaq, Original Article Link.