Basis Lab

Answers · How models work

Why does PPO clip the probability ratio?

In shortPPO clips the probability ratio so the objective stops rewarding a policy for pushing an action further in the direction it already favours, while still penalising pushes the wrong way.

PPO clips the probability ratio because the ratio multiplies the advantage, and without a limit the objective can keep paying for the same action to become more and more likely. The clip places a band around one; outside that band the objective stops increasing, so the optimiser has no further reward for pushing that action's probability in the same direction.

rt(θ)=πθ(at∣st)πold(at∣st),Lclip(θ)=Et[min⁡(rt(θ)A^t,  clip(rt(θ),1−ϵ,1+ϵ)A^t)]r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\text{old}}(a_t\mid s_t)},\qquad L^{\text{clip}}(\theta)=\mathbb{E}_t\left[\min\left(r_t(\theta)\hat{A}_t,\;\mathrm{clip}\left(r_t(\theta),1-\epsilon,1+\epsilon\right)\hat{A}_t\right)\right]

For one sampled action take a clip width ϵ=0.2\epsilon=0.2, so the band is [0.8,1.2][0.8,1.2]. If the action had positive advantage A^t=2\hat A_t=2, at rt=2r_t=2 the unclipped term is 2×2=42\times2=4. The clipped ratio is 1.21.2, so the clipped term is 1.2×2=2.41.2\times2=2.4, and the minimum is 2.42.4. At rt=3r_t=3 the unclipped term is 66 but the clipped term is still 2.42.4, so the minimum is still 2.42.4. The objective has gone flat in rtr_t: making a good action even more likely than 1.21.2 times its old probability buys nothing further in this term.

The same band is deliberately not symmetric in its effect. Take A^t=−2\hat A_t=-2, a bad action. At rt=2r_t=2 the unclipped term is −4-4, the clipped term is 1.2×(−2)=−2.41.2\times(-2)=-2.4, and the minimum is −4-4, the more negative one. So a bad action that has been pushed far above the band is still penalised at full strength; clipping alone would have softened that penalty to −2.4-2.4 and left a zero gradient at the edge. At rt=0.6r_t=0.6 the unclipped term is −1.2-1.2, the clipped term is 0.8×(−2)=−1.60.8\times(-2)=-1.6, and the minimum is −1.6-1.6. For rt<0.8r_t<0.8 that value stays at −1.6-1.6, so a bad action made much less likely also stops earning further reward. The flat side has moved to the lower edge because the advantage is negative.

What the band does not do

The natural guess is that clipping puts a hard limit on how far the policy can move. It does not. The ratio rtr_t is still whatever the current parameters make it; nothing in the objective prevents rtr_t from being 33 or well below the lower edge. What is limited is the objective's payment for that sample. On the side where the action is already being made more likely and the advantage is positive, the gradient with respect to rtr_t is zero once rt>1+ϵr_t>1+\epsilon, so an ordinary optimiser has no reason to go further. On the other side the full penalty remains. This is a one sided brake, not a fence.

Replacing the ratio by its clipped value without the minimum fails exactly here. For A^t=−2\hat A_t=-2 at rt=2r_t=2, the clipped-only term is −2.4-2.4 and its gradient in rtr_t is zero, so a policy that has made a bad action far too likely receives no correction from this term. The minimum restores the unclipped −4-4, whose gradient is nonzero. For A^t=2\hat A_t=2 at rt=0.6r_t=0.6, the clipped-only term is 1.61.6 and flat, so a good action made too unlikely would also stop being pulled back; the minimum keeps 1.21.2 and its gradient. The minimum chooses the smaller of the two terms, and that choice is what places the flat region on the safe side for each sign of the advantage.

Where it stops

Clipping is a first order imitation of a trust region, not the trust region itself. The constrained problem maximises the reweighted objective subject to an average divergence budget, and its solution is the gradient rescaled by the curvature of that divergence. Clipping has no second order object and no explicit budget on the average divergence; it only removes the incentive for further movement in the direction that is already paying. A policy can still move far if many terms in the batch push the same parameters, and the guarantee is local to the states in the batch. The old policy's state distribution is fixed, so states it did not visit get no correction at all.

The clip width sets how much of a single sample's ratio is allowed to matter. With ϵ=0.2\epsilon=0.2 the band is [0.8,1.2][0.8,1.2]; a smaller band makes the objective flatter sooner and allows less reuse of one batch before the old ratios are too stale. A larger band lets the same batch be pushed harder, at the cost of the policy moving further from the behaviour that produced the advantages. The safe range is empirical, because the objective is an approximation to the constrained problem rather than the constrained problem itself.

Worth keeping

  • Clipping bounds the objective's incentive, not the policy's actual probability ratio.
  • The minimum is essential: clipping alone would remove the correction for bad actions driven too far.
  • A smaller clip width makes the update more conservative; a larger one allows more reuse of the batch.

Where this goes next

This question is one step of Learning From Reward, a course that works through its subject from the beginning. The page that teaches this part of it is Keeping the Step Small Where It Matters, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. Why is the REINFORCE gradient so noisy even though it is unbiased?Because unbiased only fixes the average, while one noisy return is multiplied by every action's score, so the estimate's spread grows with episode length and reward scale.
  2. What is the KL penalty against the original policy actually protecting against?It stops the policy from chasing high scores in parts of the answer space where the fitted reward is only guessing, by charging it for moving away from the model it started from.
  3. Why does the value of a state have to satisfy a recursive equation with the next state's value?A state's value equals the reward from the next step plus the discounted value of wherever that step lands, because every possible future divides that way and averaging over the first step leaves exactly the next state's value.
  4. How do human pairwise comparisons turn into a reward function?Human pairwise preferences become a reward by fitting a scoring model so that preferred responses score higher, then using that score with a penalty for drifting from the starting model.
  5. Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
  6. Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.

Read alongside