Can You Really Understand KL If You Only Know Multiplication?

Ask an LLM a hundred times How long should I boil an egg?, and the answers will not be the same a hundred times. The model samples every word, so the answers spread out. In the 6th post of the post training series, Dongxi 东锡 NLP uses the boiled egg example to break KL divergence into three steps: ratio, log, and averaging by the new model's proportions. The promise at the start of the article is direct: you only need to know multiplication. Someone in the comments saw the title and said, 'Can you understand KL if you only know multiplication? I need to read this carefully.'

Five Answers, One Set of Probabilities

Dongxi gives a starting distribution like this: the vague answer Boil it until it's done. accounts for 40%; About 10 minutes. accounts for 25%; the one that first gives a long explanation of egg facts and only at the end says 10 minutes accounts for 15%; the answer that gives soft-boiled 6 to 7 minutes, hard-boiled 10 to 12 minutes, and then cooling in cold water accounts for 12%; the wrong answer that says 2 minutes is enough for a hard-boiled egg accounts for 8%. The proportions add up to 100%. This set of proportions is a probability distribution, describing the model's current habits. The starting model is called reference, the reference model.

After training for a short while, the vague answer drops from 40% to 30%, the best answer rises from 12% to 22%, and the other three answers stay unchanged. How much has the model changed? We need a number: 0 when unchanged, larger when it changes more. KL is the common measure, from the 1951 work of Solomon Kullback and Richard Leibler. Post-training cannot do without it, because the reward model is only reliable near answers it has seen during training, and RL training must make the model pay a price for deviating from the reference model.

Step 1: Ratio

For each answer, divide the new proportion by the old proportion. The vague answer is 0.30 / 0.40 = 0.75, the best answer is 0.22 / 0.12 = 1.83, and the three unchanged answers are all 1. The ratio says how many times more likely this answer has become. Greater than 1 means up, less than 1 means down, equal to 1 means unchanged.

We don't use subtraction here. The vague answer and the best answer both moved by 0.10, and subtraction cannot see the difference. The vague answer lost a quarter, while the best answer almost doubled; the ratio can show this difference.

Step 2: Take log

Ratios have an inconvenient aspect. Doubling is 2, halving is 0.5; the changes are equal in size and opposite in direction, but 2 is 1 above 1, while 0.5 is only 0.5 below 1. No change corresponds to 1, which is also not convenient. log straightens these things out: no change is 0, up is positive, down is negative, doubling is +0.693, halving is -0.693. Becoming 10 times is +2.303, becoming one tenth is -2.303.

log also turns multiplication into addition. Doubling twice equals multiplying by 4, log(4) = 1.386, exactly 0.693 + 0.693. As the ratio approaches 0, log keeps decreasing without a lower bound. log(0) is undefined.

Step 3: Average by the New Model's Proportions

Each answer now has a log ratio, and we need to combine them into one number. The method is a weighted average, with weights given by the new model's proportion for that answer. The vague answer term is 0.30 × (-0.288) = -0.086, the best answer is 0.22 × (+0.606) = +0.133, and the other three answers are 0. The sum is 0.047. This is the KL of the new model relative to the reference model, in units called nats.

The formula is written as:

KL(π ‖ π_ref) = Σ π(y) log(π(y) / π_ref(y))

π is the model being trained, and π_ref is the reference model. Ratio, log, weighting by the new model's proportions, and summing: every piece of the formula has appeared above. The weights use the new model's proportions because we care about the new model's current behavior. Answers it says often count more; answers it rarely says count less.

What This Number Is Saying

log ratio has a concrete meaning. Suppose someone shows you an answer and asks you to guess which model it came from. If it is the best answer, the ratio is 1.83; the new model writes it 1.83 times as often as the reference model, so this answer is evidence for the new model. If it is the vague answer, the ratio is 0.75, and it is more like evidence for the reference model. The log ratio is the size of the evidence. When looking at multiple answers, evidence can add up. KL can be understood as: if the answers really come from the new model, how much evidence pointing to the new model each answer provides on average.

KL 0 means the two models are exactly the same. KL 0.047 means each answer provides very little evidence; to reach 20-to-1 confidence, you need log(20) = 3.0 evidence, so on average you need to look at about 64 answers. A model that always gives the best answer has KL 2.12; a single answer already gives 8.33-to-1 confidence.

Why KL Cannot Be Negative

A single log ratio can be negative, but KL cannot. The proportion by which one answer decreases is the proportion by which another answer increases; the total change adds up to 0. Each term of KL is no less than that answer's change. The vague answer's change is -0.10, and this term is -0.086; less is subtracted than it lost. The best answer's change is +0.10, and this term is +0.133; more is added than it gained. The total change is 0, and each term is no less than the corresponding change, so KL cannot be less than 0. This property comes from the shape of the log curve; the log curve always lies below the line x - 1 and touches it only at x = 1.

The weights also need to be used correctly. If you add the log ratios directly without weighting, you get a negative number. If you switch the weights to the reference model's proportions, you also get a negative number. Only KL weighted by the new model's proportions gives a usable reading. The model changed, but the distance is negative; that makes no sense.

KL Only Cares About Distance; Direction Comes From Reward

That change just now made the model better, and KL is still positive. KL does not read the content of answers and does not know which answer is good. Moving 0.10 toward the best answer costs 0.047; moving 0.10 away from it costs 0.076. Both movements must be paid for. Direction comes from reward.

The total score RL training gives the model is average reward minus the price of KL:

Total score = average reward - β × KL

β is the price of one unit of KL, decided by the people doing the training. Reward decides the direction of movement, and β × KL limits the distance moved. The higher the price, the less the model dares to leave the reference model.

Same Movement, Different Prices

Add a sixth answer: Boil it 20 minutes, always, for food safety. Experts agree. It is wrong, but its tone is very confident. The reference model almost never writes it, only 0.5% of the time. Suppose the reward model has rarely seen this kind of answer and gives it an overly high score.

Compare two changes, both taking 0.10 away from the vague answer. Giving it to About 10 minutes changes 25% to 35%, the ratio is 1.4, and KL is 0.031. Giving it to that confidently wrong answer changes 0.5% to 10.5%, the ratio is 21, and KL is 0.233. The proportion moved is 0.10 in both cases, but KL charges about 7.4 times as much for the second change. Because what enters the formula is the ratio.

This is exactly what training needs. For answers the reference model almost never writes, the reward model has also rarely seen them, and its score is least reliable there. KL charges the most precisely where the reward signal is weakest.

KL Is Asymmetric

Swap the two models in the formula and switch the weights to the reference model's proportions, and you get KL(π_ref ‖ π). This small change computes to 0.042, while the original direction is 0.047. KL is asymmetric, so it is called divergence, not distance.

The order determines which kind of change is expensive. If the new model stops saying the wrong 2-minute answer and gives its 0.08 to the best answer, under the RLHF order KL is only 0.102. After swapping, the reference model still says this answer 8% of the time, while the new model never says it; the ratio is infinite, and KL is also infinite. Newly added answers are the opposite. RLHF uses KL(π ‖ π_ref), so the model can confidently drop bad answers, while adding answers that the reference model almost never says is very expensive. Cross-entropy in pretraining equals data entropy plus KL(data ‖ model); the data is on top, so the model must leave some probability for everything that appears in the data.

Real Models: Samples and Tokens

A real model may be able to write too many answers to list. KL is an average over all answers from the new model, and the average can be estimated with samples. Let the new model write a batch of answers, compute each log ratio, and then average. Simulating the egg example once: 10 answers give 0.096, 100 answers give 0.061, 10000 answers give 0.048, and the exact value is 0.047. Samples must come from the new model itself. If you lower temperature or add top-p, the estimate is no longer this KL.

The probability of an answer is the product of the probabilities of each token. log turns the product into a sum, so the log ratio of an answer is the sum of the log ratios of each token. The log ratios of three tokens can be 0.223, 0, and 0.182, summing to 0.405. KL can be computed token by token, requiring only one extra forward pass through the frozen reference model. InstructGPT's PPO subtracts β × log ratio from the reward at each token, with β = 0.02. DeepSeekMath's GRPO adds KL into the loss, with β = 0.04; its estimate for each token is x - log x - 1, where x = π_ref / π.

How to Read KL in Training Logs

Step 0 must be 0. The model and the reference model have the same weights; a non-zero reading means there is a bug. A batch average that is clearly negative is usually a bug too. A single token can be negative, but if the average stays below 0, common causes are that the sampling settings and the probability-computation settings do not match, or that the mask includes prompt and padding tokens. Pay attention to the units. KL per token and KL per answer differ by the answer length; only runs with the same units and the same prompts can be compared.

There is no so-called good value for KL. It only says how much the model has moved. Whether this movement is worthwhile depends on signals the training has never seen: human comparisons on held-out data, benchmarks, or a verifier on checkable tasks. This independent signal is called the gold signal. In Dongxi's toy example, as KL increases, reward keeps rising, while gold peaks at 1.10 at KL 0.81 and then declines. At KL 3.61, reward is very high, while gold has already reached -0.76. Gao, Schulman, and Hilton reported this pattern in 2022 with a real reward model. KL should be read together with reward and gold signal; the checkpoint to keep is the one where gold peaks.

log ratio Is Also a Kind of Reward

log ratio says how much the model raises or lowers each answer. When the model raises an answer, it looks like it values that answer. In the egg example, at the compromise point, every log ratio equals reward minus the same constant 0.808. When comparing two answers, this constant cancels out. For the best answer versus About 10 minutes, using reward gives a difference of 2 - 1 = 1, and using log ratio also gives 1.192 - 0.192 = 1; Bradley-Terry gives the same probability.

This is the idea of DPO. It omits a separate reward model; for each pair of answers chosen by humans, it uses β × log ratio as the score and plugs it into the Bradley-Terry loss. The starting model's log ratios are all 0, the score difference is 0, and the loss is 0.693; at the compromise point the score difference is 1, and the loss is 0.313. The log ratio in KL measures how much an answer has moved, and it can also be treated as the reward for that answer.

Summary

KL uses one number to compare two answer distributions. The algorithm has only three steps: divide the proportions, take log, and average by the new model's proportions. It cannot be negative, and it is 0 only when nothing has changed. It has no direction; direction comes from reward, and distance is priced by β × KL. It charges the most for raising answers that the reference model rarely writes. It is asymmetric; the RLHF order lets the model drop answers, while newly added answers are very expensive. On real models it is estimated by sampling, and it can be computed token by token. During training, don't look only at KL; read it together with reward and gold signal.

Original article and video from Dongxi 东锡 NLP: Post training series post 6, article link.