The RL sampling mismatch table is comprehensive, but the replies point the finger at serving

On X, Nathan Lambert was going through the score-centering RL paper, planning to add notes for his book, and came across a table in the appendix. The table, called Table 1, arranges train-inference mismatch correction methods based on importance sampling into a grid.

This table is very dense with information. Level determines whether the statistic is computed per token or per sequence. Stat is ratio or tv. Low and High give the interval, and Outside says whether values beyond the interval are clipped or dropped. Sign says which advantage signs are corrected: both weights every token, neg weights only negative advantages, and ppo uses PPO's pessimistic clip, binding only in the direction that would increase the objective.

The table lists Naive IS, TIS, CISPQ, MIS (i.e., IcePop), PPO, DAPO, GSPO, TOPR, and DPPO. A few details are worth noting: PPO and DAPO both use ratio at the token level, with intervals of 0.8 to 1.2 and 0.8 to 1.28 respectively; values outside are dropped; Sign is ppo; and they include dual clipping at 3. GSPO jumps to the sequence level, with a ratio interval of 0.9997 to 1.0004; values outside are dropped. TOPR is at the sequence level, with an interval of 0 to 1.0; values outside are clipped; it corrects only negative advantages. DPPO uses tv, with an interval of 0 to 0.2; values outside are dropped; it does not do importance-ratio clipping.

Nathan's assessment is that since GRPO took off, these methods have become one of the biggest focal points in RL algorithm research. This table works well as an index, making it quick to see each method's trade-offs across token vs. sequence, clip vs. drop, and which signs are corrected.

The replies did not stop at the algorithm layer. Mohammad Vazin said that half of the mismatch is a serving problem dressed up as RL theory. Rollout engines and trainers use different kernels, and batch-dependent reductions and accumulation orders also differ; even with the same weights, logprobs will drift. His suggestion is direct: first fix deterministic rollout kernels, then talk about IS techniques.

Renan Santos divided the methods in the table into two categories: TIS-style token-level truncation, which directly clips the ratio; and masking out tokens or sequences whose sampler and trainer logprobs deviate too much. Clipping preserves the biased signal for every token, while masking loses data but is closer to on-policy. This trade-off is closer to the training reality than the clip and drop in the table.

Max Petrusenko added an engineering precondition: importance weights are only useful when the difference between sampler and trainer probabilities is recorded per token. Otherwise the mismatch hides in reward noise; the weights look like they are correcting something, but in practice there is nothing traceable to act on. Saksham Jain explained the GSPO numbers that look like a typo: compared with PPO's 0.8 to 1.2, the 0.9997 to 1.0004 range is because GSPO takes a geometric mean over the entire response, so it is no surprise that the statistic is close to 1.

Cameron R. Wolfe posted a related blog in the replies: Off-Policy Drift in LLM RL. Gangadhar Payyavula mentioned that score centering gave a clear improvement when he trained decision models, and he was a bit surprised by the eval results.

This table can easily be mistaken for an algorithm selection checklist, but it only covers the IS correction layer. If the logprobs of the rollout and trainer are themselves drifting, weight correction is just dressing up system noise. IS techniques are elegant, provided that probability differences are recorded and differences in kernels and reductions are controlled. The replies make this point very clear: fix serving first, then talk about algorithms.