A common activation function bug has existed for years—the fix is just subtracting a constant

Ziming Liu at MIT pointed out on Twitter/X a “bug” in commonly used activation functions: the outputs of ReLU, SiLU, and GeLU do not have zero mean.

This sounds like a triviality. ReLU cuts off the negative half-axis, so its output is of course skewed positive. But Ziming Liu’s paper treats this “triviality” as a serious problem, and after fixing it, improvements were seen on LLMs, ViTs, and DiTs.

Where the bias is

Today’s most widely used activation functions are asymmetric: the negative half-axis is suppressed more strongly than the positive half-axis. Even if the input distribution is centered at zero, after passing through the activation function the mean of the output shifts in the positive direction. The figure in the original paper makes this clear: a distribution that was originally centered moves to the right as a whole after passing through an unshifted activation function.

The impact is significant

This bias looks harmless, but it makes the loss landscape ill-conditioned. Concretely, it manifests as loss spikes in late training, representation overshoot, and slower convergence.

The fix

Just subtract a constant:

ReLU  -> ReLU - 0.2
GeLU  -> GeLU - 0.115
Swish -> Swish - 0.077

That’s it.

In early training, the loss after subtraction is slightly higher, but it quickly catches up and overtakes. Later, the loss is lower and the curve is more stable.

Representation evolution is also smoother. The trajectory with bias suddenly turns midway; once the bias is removed, it becomes smooth and monotonic.

The effect is even clearer in an autoencoder: after removing the bias, the representation appears to evolve monotonically.

From the perspective of the loss landscape, removing the bias makes the eigenvalues noticeably smaller. Optimization is no longer dragged down by sharp “valley” directions, and it is less prone to oscillation and spikes.

Results

There were improvements in all three settings: image classification with ViT, image generation with DiT, and language modeling with GPT. Ziming Liu said in the tweet, “This useful trick is free.”

Paper and background

Paper link: https://arxiv.org/pdf/2610.14500

This is the fourth day of content from MetaCircle AI Research Exposition. When Dan Roy retweeted it, he mentioned that mufan_li et al. used a proportional limit model to document and study the relevant pathology.

One observation

The interesting thing about this finding is that it does not reveal any new mechanism. Everyone doing deep learning knows that activation function outputs do not have zero mean, but knowing it is one thing; few people treat it as a defect that must be fixed. What Ziming Liu did is roughly equivalent to taking a design tradeoff that everyone had accepted by default, treating it as a bug again, fixing it, and then showing that fixing it really does pay off.