280B with Only 14B Active, ERNIE 5.5 Flash Base Open-Sourced: SOTA Has a Footnote

Baidu has released the weights for ERNIE 5.5 Flash Base. It has 280B total parameters, activates 14B per token, was pretrained on 33.5T tokens, with a training context of 128K, and supports 1M for fine-tuning and inference. The release card is clear: this is the base checkpoint at the end of pretraining, with no mid-training or post-training. For developers, it is suitable for continued domain adaptation, instruction fine-tuning, and reinforcement learning research. For ordinary users, it is more like an engine than a complete car, and cannot be used directly as a chat assistant.

Parameter utilization is the most noteworthy part this time. Of the 280B, only 14B is activated per token, so the sparsity ratio is not low. The official architecture has 43 layers: 1 dense FFN layer plus 42 MoE layers. Each MoE layer has 512 routed experts, with top-10 activation, plus 1 shared expert; experts perform dispatch and combine in a 2048-dimensional latent space. Attention is 37 layers of HCA plus 6 layers of MLA-DSA. HCA caches one entry every 128 tokens, saving KV cache. MLA preserves full-context access. The official claim is that under a similar KV cache budget, HCA with MLA performs better than with CSA. This design breaks down long-context cost separately rather than focusing only on activated parameters.

In replies, the official account added several design choices. Multi-Router Latent MoE uses multiple independently initialized sigmoid routers to vote, reducing the weak correlation between a single routing score and the actual usefulness of experts. ECHO places multi-token prediction at the last layer of the backbone; this layer also serves as the draft head for speculative decoding, saving a separate MTP module. There are also mHC, a MultiMax output head, and a BPE tokenizer with a vocabulary of about 200K, with numbers split from right to left in groups of at most three digits. On the training side, they mention a distributed Muon optimizer and the FlashMask-4 attention kernel, aiming to keep training efficiency under control at the scale of thousands of GPUs.

The evaluation table needs its caveats. The official setup uses a unified pipeline, prompts, and few-shot settings, comparing against DeepSeek-V4-Flash-Base 284B-A13B, Hy3-preview-Base 295B-A21B, and MiMo-V2.5-Base 310B-A15B. ERNIE 5.5 Flash Base scores 93.3 on MMLU, versus 89.4, 87.5, and 87.6; 78.4 on MMLU-Pro, versus 69.2, 66.9, and 65.7; 82.3 on HumanEval+, versus 69.5, 78.0, and 74.4; 40.1 on LiveCodeBench, versus 31.9, 33.5, and 32.4; 91.7 on RepoQA, versus 81.0, 91.3, and 56.5; and 48.7 on LongBench-v2, versus 44.5, 39.8, and 44.6. It is not a clean sweep. ChineseSimpleQA is 73.0, below DeepSeek’s 75.3; INCLUDE-lite is 79.5, slightly below 79.8; FullStackBench is 54.6, below Hy3’s 56.7; HELMET-Rerank is 56.2, below DeepSeek’s 60.0. Its strengths are general knowledge, code, and long-context retrieval, and its weaknesses are also written in the table.

The word “SOTA” is often used very broadly in model cards. The official claim is same-size SOTA, and the comparison list is indeed selected by total and activated parameters. But commenters were not convinced: some said the comparison targets looked like models from six months ago, some directly asked why newer models were not included, and others asked about vision capabilities and minimum GPU deployment configurations. Some also noticed that 280B activates only 14B, and argued that inference cost could be reduced considerably. These reactions are more useful than slogans. After open-sourcing, deployment cost, fine-tuning results, and real-task performance will give harder answers.

The weights are available on Hugging Face and ModelScope, in FP8 and BF16 versions, under the ERNIE 5.5 Flash Base Model License v1.0. SGLang and vLLM both provide deployment methods, and long-context inference can go up to 1,048,576 tokens. The official full version of ERNIE 5.5 Flash is said to still be on the way. The current Base is suitable for secondary training and research, not for directly treating benchmark numbers as product conclusions.

Links:

• Hugging Face model page
• Technical blog
• ModelScope
• Original X post