How Does One Score Get Distributed Across Thousands of Tokens? Using “Is OpenAI open” to Explain RL Terms Thoroughly

Dongxi 东锡 NLP recently wrote an article about LLM reinforcement learning terms. The title is straightforward: RL terms for LLMs. The difficulty is also straightforward: policy, state, action, reward, return, value, advantage, baseline, TD error, GAE, clipping, KL—easy to skim past as a vocabulary list, but hard to remember once they are put into formulas. The article makes a choice: it threads these terms together with one example that runs through the whole piece.

First, some background. In January 2025, DeepSeek released DeepSeek-R1-Zero, which did RL directly on the base model without first doing supervised fine-tuning. The reward comes from rules: one checks the final answer, and the other checks whether the reasoning process is written between think tags. After training for a few thousand steps, the model's pass@1 on AIME 2024 rose from 15.6% to 71.0%, and its answers also became longer. The rules give only one score for the entire answer, but training needs to adjust the probability of every token. That is where the problem lands: if the judge scores only the whole answer, which tokens' probabilities should be increased, and which should be decreased?

The tweet includes a short illustrative video:

The article's example is the question Is OpenAI open? The name says open, and the answer may be partly open. The model has already written OpenAI is, and now it chooses the next token. It most wants to choose open, with probability 0.5; the answer continues with It's in the name., and the judge gives an average score of 2. closed has probability 0.3, misses open-weight models, and gets 5. partly has probability 0.2, continues with some open weights, and gets 9. This example is small, but it is enough to ground each term one by one.

The agent is the model. The environment is the prompt and the judge. The judge can be a reward model trained on human preference data, the rule checker from R1-Zero, or unit tests or another LLM that scores according to criteria. During RL training, the reward model is frozen, so the judge counts as the environment. The model's probability distribution over the next token is called the policy, denoted π_θ, where θ is the weights. Training changes θ, and therefore changes the policy.

Within an answer, everything proceeds token by token. The state is all the content seen before writing the next token: the prompt plus the tokens already written. The action is the next token. The policy assigns probabilities to candidate tokens, samples one, appends it to the state, and repeats until the answer ends. The complete answer is called a trajectory, also called a rollout or an episode.

Reward is the score at each step. In LLM RL, every token except the last has reward 0, and the last token gets the judge's score; this is called sparse reward. Return is the sum of rewards from a certain token to the end. General RL has a discount factor γ. LLM answers end, and the score belongs to the whole answer, so γ=1 is used. In the example, the return at every position equals 2. Looking only at return, you cannot tell which token is good; the later terms are about distributing this score onto tokens.

Value, Q, advantage. Go back to the state OpenAI is and let the model continue writing many times. Among ten samples, the five that start with open get 3, 1, 2, 1, 3, so on average Q(open)=2; the three that start with closed get 5, 4, 6, so on average Q(closed)=5; the two that start with partly get 10, 8, so on average Q(partly)=9. The average of all ten scores is V=4.3. Advantage A=Q−V: open is −2.3, closed is +0.7, and partly is +4.7. Training actually uses advantage. The sign gives direction, and the magnitude gives size; it is relative to the current model. The probability-weighted average is 0, so one update can only move probability from below-average tokens to above-average tokens.

Baseline. In actual training, you cannot get Q; you can only see sampled answers and scores R. You estimate advantage by subtracting a baseline from R, and the baseline is usually V. For the five open answers, R−V is −1.3, −3.3, −2.3, −3.3, −1.3, which averages to −2.3, exactly Q−V. The baseline does not change the update direction; it changes the noise. In the article's simulation, using V as the baseline gives about one fifth the variance of using 0 as the baseline. GRPO's within-group average score and PPO's critic are baselines in this sense.

Credit assignment. A real answer has many tokens, and each token needs its own share. TD error measures the change in expected score at each token: δ=r+V(next state)−V(current state). In the example, V drops from 4.3 to 2 after open, and the TD error is 0+2−4.3=−2.3. The responsibility is assigned to open. GAE introduces λ. When λ=0, each token's advantage is its own TD error, and all responsibility lies with open. When λ=1, advantage is the final score minus that token's V, and OpenAI, is, and open all receive −2.3. PPO often uses λ=0.95. GRPO has no per-token V, and all tokens in one answer share the same advantage.

Policy gradient. Tokens with positive advantage increase their log probability, and tokens with negative advantage decrease it, with the adjustment proportional to the advantage. In the article, after two expected-gradient updates, the probabilities go from 0.50/0.30/0.20 to 0.15/0.36/0.49, then to 0.04/0.11/0.86, and V rises from 4.3 to 6.5 and then to 8.3. closed's probability rises in the first step because 5 is above 4.3; after V exceeds 5, closed is below average, and its probability starts to fall.

Exploration. Greedy decoding always chooses only the highest-probability open, with average score 2. The advantage of open is 0, nothing changes, and it will never discover that partly can get 9. With temperature-1 sampling, the model sometimes writes closed or partly. These samples show that other branches score higher, and the policy changes based on them.

Small-step updates. Ratio and clipping: PPO uses the current model to sample a batch of answers, saves a frozen copy π_old, and updates on the data for a few steps. The importance ratio ρ=π/π_old measures the probability change and adjusts the weight. If ρ goes outside [1−ε,1+ε], updating stops for that token. With ε=0.2 and π_old(partly)=0.2, an advantage of +4.7 can push π(partly) to at most 0.24. KL penalty: if the judge likes flattery, it gives partly. What a brilliant question! 9.6, and the reference model has a 5% probability of writing the flattering version. Without constraints, RL would push the policy toward flattery. The KL penalty subtracts β×KL(π‖π_ref) from the reward. With β=1, the flattery proportion rises from 5% to 90%, the judge's average score only rises from 9.03 to 9.54, the KL term grows to 2.38, and R−β×KL drops from 9.03 to 7.16. This is a small example of reward hacking; KL keeps the policy near the reference model.

Summary. The article draws the entire loop in one diagram and also provides a quick-reference table. The LLM is the policy, the prompt plus already-written tokens is the state, the next token is the action, and the complete answer is the trajectory. The judge scores only the finished answer. V is the average score starting from a state, Q is the average score after writing a certain token, and advantage A=Q−V. Baseline reduces noise. TD error and GAE assign credit. Policy gradient adjusts probabilities. Exploration provides samples. Ratio, clipping, and KL limit the size of updates. The next article will cover preference alignment: how a reward model is trained from human preference data, and how PPO, GRPO, and DPO use it.

Original article link: https://x.com/i/article/2115385031132635138

Tweet link: https://x.com/dongxi_nlp/status/2115390983818944513

The substance of this piece is not laying out a vocabulary list. It breaks “how scores land on tokens” into a line you can follow. RL terms are still hard, but at least now there is an example you can retell.