Answers · How models work

What is the KL penalty against the original policy actually protecting against?

In shortIt stops the policy from chasing high scores in parts of the answer space where the fitted reward is only guessing, by charging it for moving away from the model it started from.

It protects against the policy moving into responses where the fitted reward is extrapolating, not against the fitted reward being wrong in general. The penalty makes every departure from the starting model pay, so a response has to earn more fitted reward than the charge for becoming likelier than it was.

\mathbb{E}_{y \sim \pi_\theta}\left[r_\phi(x,y) - \beta \log \frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y\mid x)}\right] = \mathbb{E}_{y \sim \pi_\theta}[r_\phi(x,y)] - \beta \, \mathrm{KL}(\pi_\theta(\cdot\mid x)\,\|\,\pi_{\text{ref}}(\cdot\mid x))

Take $\beta=2$ . A response with probability $0.01$ under $\pi_{\text{ref}}$ that the policy lifts to $0.02$ has log ratio $\ln(0.02/0.01)=\ln 2=0.693$ . The charge is $2\times 0.693=1.386$ . If the fitted reward for that response is $3.0$ , the net reward is $3.0-1.386=1.614$ , so the policy still moves. At $\beta=5$ the charge is $5\times 0.693=3.465$ and the net is $3.0-3.465=-0.465$ , so that response is no longer worth it. The break-even $\beta$ is $3.0/0.693=4.33$ .

FIG 1

beta	charge	net reward	does it move?
0	0.000	3.000	yes
1	0.693	2.307	yes
2	1.386	1.614	yes
5	3.465	-0.465	no

The natural guess

The natural guess is that the term keeps the policy close to where it started, full stop, or that it regularises the learned reward. Keeping close is the mechanism, not the purpose: if the reward were true, moving away would be exactly what you want, and the term would only cost you reward. It does not regularise the learned reward either. The fitted reward $r_\phi$ is held fixed; the charge is on the policy's departure, not on the reward model's parameters. Nor does it stop reward hacking in the abstract: it prices exploits by how much drift they require. An exploit already available at the starting model costs nothing to reach.

The charge accumulates over tokens. If a ten-token response has every token's probability doubled, the log ratio is $10\ln 2=6.93$ , so at $\beta=2$ the charge is $2\times 6.93=13.86$ . A fitted reward would need to exceed $13.86$ just to break even. In one sample the term is just the log ratio; in expectation it is the KL divergence. That is why $\beta$ is the one dial: it sets the exchange rate between fitted reward and distance from the starting model.

Where it stops protecting

The protection is only as good as the region it enforces. If $\beta$ is zero, there is no charge and the optimiser runs to the fit's mistakes. If $\beta$ is small relative to the reward scale, the break-even log ratio is large, and a response only needs a modest fitted reward advantage to justify substantial drift. If $\beta$ is too large, the charge exceeds every available reward gain and the policy does not move. If the fitted reward is already wrong near the starting model, the tether holds the policy to a bad target. And if the starting model has a high-probability hedge that the reward model likes, the penalty does not remove it, because staying there costs nothing.

What is protected

What is being protected is the meaning of the fitted score, not the quality of the answer directly. The reward model was fitted on responses like the starting model's, so its ranking is informative there and arbitrary outside. The KL term converts 'do not leave that region' from a hard constraint into a price, and the policy leaves only when the fitted reward says the outside is better by more than the price. That is the whole protection: it makes the optimiser pay for the extrapolation it is about to trust.

Worth keeping

The KL penalty is a price on policy drift, not a correction to the reward model.
A fitted reward is trustworthy near the responses it was trained on and unreliable elsewhere, so the penalty keeps optimisation inside that trustworthy region.
Set the price too low and the optimiser exploits the fit; set it too high and the policy does not move.

Where this goes next

This question is one step of Learning From Reward, a course that works through its subject from the beginning. The page that teaches this part of it is When the Reward Is Itself Learned, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course