Score Centering Enters Tinker's Stack, Putting RL Training-Inference Drift on the Table

Nathan Lambert said in a tweet that score-centering is no longer a niche idea in industry. Tinker's StableReinforce is using it, and it has recently been added to the RLHF book. He mentions that this derivation is useful for understanding training-inference mismatch, and a video explanation will be out soon.

This is worth watching because RL training has a long-tolerated crack: the sampler and the trainer follow two different numerical paths. The sampler generates rollouts, and the trainer updates parameters. Quantization, kernels, parallelism strategies, and version differences can all make the probabilities of the same piece of text not exactly consistent across the two engines. This is TIM. Completely eliminating TIM is costly, and rollout efficiency would suffer.

Martin Marek's explanation is more direct: at every training step, the trainer is pulled toward the sampler, and the bias accumulates step by step, forming drift. What Score Centering does is subtract the expected score from each score. In the on-policy case, the expected score is zero, so this step is a no-op; in the off-policy case, it cancels the drift term and prevents the trainer from quietly distilling the sampler.

On overhead, Martin's implementation uses a top-k approximation layered on top of an arbitrary importance ratio, written as a scalar loss. The premise is that the sampler-to-trainer path is already transmitting top-k logprobs; in that case, the added overhead is below 1%. For online RL systems, this number matters more than the formula, because it determines whether the correction term can live permanently in the training loop.

The paper is titled Score Centering Stabilizes Off-policy Reinforcement Learning, by Martin Marek and Max Ryabinin, and the work was done during Martin's internship at TogetherAI. The conclusion in the abstract is clear: RL is unstable under TIM, and this mainly comes from drift. Score Centering is an additive correction used to cancel accumulated bias. Across model sizes from 0.6B to 30B, under quantization, score centering alone matches or exceeds importance-sampling-based methods, and the more severe the mismatch, the larger the gap. It can also be combined with importance sampling; in staleness experiments, the combined approach outperforms the pure IS baseline.

What is more noteworthy is the speed of adoption. The code repository lists implementations including verl, TRL's AsyncGRPOTrainer default configuration, slime, Miles, prime-rl soon to be default, Molt, SkyRL, Vime, and Tinker. One correction term entering so many training stacks at once shows that the pain point it addresses is common enough. The repository provides JAX and PyTorch loss implementations, with top-k set to k=128, and the tail scaled by the trainer distribution to match the sampler tail mass.

The discussion also includes an opposing view worth reading. Nan Jiang said on X that, considering a linear softmax policy, this correction amounts to computing the gradient under the sampling distribution, and can be viewed as an implementation that ignores mismatch. You Jiacheng replied that it still removes drift, and the resulting gradient is exactly the on-policy policy gradient of the sampling policy. Nan Jiang then clarified that he sees this as one of many reasonable implementations, and the effect is to ignore mismatch. This debate lays out what Score Centering actually changes, and is more informative than just looking at training curves.

There is also a question with no clear answer. Max Petrusenko asked whether score centering still works when the sampler changes during training, or whether it only applies when the mismatch between sampler and trainer stays stable. The original post did not provide an answer. If the sampler in your RL stack dynamically shards, changes versions, or changes quantization strategy, don't rush to treat this as a universal patch; first measure drift and the actual benefit of the combined correction.

Related links: Nathan's tweet, Martin's tweet, paper, code.