The 594B-Parameter “Small Model” Drops the Linear-Attention Hybrid Layer

Sebastian Raschka posted observations about Inkling 2 Small on X. He noted that the model is more interesting than he initially thought. The reason isn't the score; it's the architecture: it uses attention residuals, similar to Kimi K3, but without a linear-attention hybrid.

This terminology needs a couple of clarifications. Attention residuals can be understood as keeping an information pathway between layers, allowing deep networks to transmit and mix representations more stably. A linear-attention hybrid mixes linear-attention layers into the backbone to reduce the computational cost of long contexts. Kimi K3 combines both; Inkling 2 Small keeps only the former. One commenter said this combination looks like a natural ablation experiment: if the scale is comparable, one can break down where Kimi K3's gains come from—whether from the residual technique or from the linear-attention mixing layer.

In the replies, someone broke down the configuration: a 594B-parameter MoE, 384 experts, 8 experts routed per token, 0 shared experts; 8 blocks, each with 6 layers, using a learned query for mixing; a 200k vocabulary. Adam Wang's remark was direct: calling a 594B-parameter MoE “small” is 2026 language habits; it scores only 31 on the Artificial Analysis index, so what matters is the architecture, not the ranking. Giacomo Masseroni focused on the choice of 8 routed and 0 shared experts; he wondered whether load balancing can still be maintained with 384 experts and no shared experts.

Questions followed. iClaude asked what attention residuals buy over a linear-attention hybrid, and which is more stable in long contexts. KURAOpenclaw asked how the complexity overhead is handled after removing the hybrid layer. Stefan Palm noticed the 200k vocabulary and wondered whether a larger vocabulary can bring direct benefits. None of these has a settled answer, but they are more useful than just reporting a leaderboard score.

DearHua's observation carries a bit of historical perspective: a few years ago, Chinese labs were copying Western architectures; now Western model release notes say “same as Kimi.” This sentence can go into a year-end architecture trend analysis. Inkling 2 Small provides a clean control: with the same residual idea but without the linear-attention hybrid layer, what price will be paid for long context and expert balance must wait for more empirical tests. Original post: https://x.com/rasbt/status/2121613812990095543