Overview
- PPO is a policy gradient method that limits how much the model can change in each update
- it compares the current policy with a frozen old policy using a probability ratio
- the ratio is clipped to a small band, so the model gets no extra credit for large jumps
- this lets one batch be reused for several updates, and the note ends with what clipping does and does not do
Proximal Policy Optimization (PPO) is a policy gradient method whose key idea is to limit how much the policy can change in each update. It lets the optimiser reuse one batch of samples for several updates without letting the model jump too far.
Step 1: freeze the policy that made the data
PPO compares every update with a fixed snapshot of the model that collected the current batch.
- old policy : generates the answers in the batch, its probabilities are recorded, and it stays fixed during these updates
- current policy : starts as a copy of , receives several gradient updates, and its probabilities change
Example: , and after one update .
The old policy is frozen so that the question "how much did we change?" always has one meaning, because the denominator below stays stable.
Step 2: measure the probability change
where:
- is the probability ratio for decision
- is the sampled choice (for example a token) at step
- is the context at step
- is the probability under the current policy
- is the probability when the example was collected
Examples:
- means no change
- means 20% more likely
- means 50% less likely
describes one sampled choice. It is not an overall quality score for the model.
At the first update on a new batch, the current policy is still a copy of the old one, so every . At that point the PPO gradient is the same as REINFORCE with advantage. The ratio starts to matter from the second update on the same batch, when the current policy has already moved.
Step 3: combine the change with the advantage
The unclipped learning signal is , where is the advantage from the Advantage Function. Its meaning depends on both signs:
- and : a good choice became more likely (wanted)
- and : a good choice became less likely (not wanted)
- and : a bad choice became less likely (wanted)
- and : a bad choice became more likely (not wanted)
PPO should reward the two wanted directions, but only up to a limited change.
Step 4: clip an oversized increase
Let and , so the sampled choice was better than expected. PPO uses .
| Ratio | Unclipped | PPO uses |
|---|---|---|
| 1.10 | 2.20 | 2.20 |
| 1.20 | 2.40 | 2.40 |
| 1.60 | 3.20 | 2.40 |
Without clipping, would keep receiving extra credit. With clipping, the incentive stops at 1.20.
Step 4 again: clip an oversized decrease
Let and , so the sampled choice was worse than expected. Lowering its probability helps, but PPO removes any extra benefit beyond a 20% decrease.
| Ratio | Unclipped | PPO uses |
|---|---|---|
| 0.95 | -1.90 | -1.90 |
| 0.80 | -1.60 | -1.60 |
| 0.50 | -1.00 | -1.60 |
At , the unclipped objective would reward the extreme drop. PPO keeps the conservative value from .
What happens without clipping
The optimiser usually reuses one batch for several epochs. That batch was generated by , not by a model far in the future.
- : the batch was just collected
- : a useful change
- : the evidence is becoming stale
- : one sample dominates
Without clipping, the objective keeps rewarding more movement on the same old examples. The possible result is unstable jumps, damaged behaviour on other inputs, and a batch that no longer represents the current model.
Complete clipped objective
where:
- is the average over the sampled decisions
- is the current probability divided by the old probability
- is the advantage, better or worse than expected
- is the clip range, for example 0.20
- forces to stay inside the band from to
- takes, for each sample, the more conservative of the ordinary value and the clipped value, so there is no extra incentive past the band
The only removes extra reward. It never softens a penalty:
- wanted direction, past the band: the clipped value is smaller, so picks it, and the extra credit stops
- unwanted direction (a good choice became less likely, or a bad choice became more likely): the unclipped value is smaller, so picks it, and the full penalty stays
- so the model is always pushed back from bad moves, but it cannot earn extra by moving too far in a good direction
Example with and , a good choice that became less likely: and the clipped value is , so keeps , and the gradient still pushes the probability back up.
The parameters move by gradient ascent:
where is the learning rate and is the update step.
One complete mini-batch
Let :
| Sample | Clipped | Used | |||
|---|---|---|---|---|---|
| A | +2.0 | 1.50 | +3.00 | +2.40 | +2.40 |
| B | -1.0 | 0.60 | -0.60 | -0.80 | -0.80 |
| C | +0.5 | 1.10 | +0.55 | +0.55 | +0.55 |
Average objective . The optimiser changes to increase this average, then recomputes the ratios.
What clipping does and does not do
Clipping does:
- flatten the sampled objective once the ratio crosses the band
- reduce the benefit of an oversized update on the current batch
- allow several mini-batch updates on the same data
Clipping does not:
- directly clamp every parameter
- guarantee an exact KL distance from the old model
- protect behaviour that is not represented in the batch
- replace the reference-model penalty used in RLHF
One huge parameter update can still change outputs that were never sampled. So the KL distance should be monitored, and the reference-model penalty should be used separately.
Note
A small change is rewarded as an improvement, and an oversized change loses its extra incentive because of clipping. PPO still learns, but no single update can overpower the rest of training.
My understanding:
PPO's purpose is to allow us to reuse each batch of samples again, for many weight updates safely.
Remember how we have offline and online rejection sampling, PPO is still mostly online, but it has some offline element to it. We are still trying to use the latest model to generate new samples for each batch, but we also reuse the earlier samples from the previous prompt on the same prompt again, in the same batch.
For example, we have 2 batches in the full RLHF. Batch 1 has prompt 1 and 2. Batch 2 has prompt 3 and 4. We first pass prompt 1 and 2 into the model. Model generates the answers, and the reward model scores the answers. At this point, we have the answers, their advantages, and the old probabilties . We then update the model which will update the current probabilties, . We then reuse the answers, the advantages and the old probabilities which is now , for the same prompts 1 and 2 in this same batch, giving us new model updates.
A possible concern is that the model might change its probability distribution too much by change its weight by an extensive amount. So this is why we introduce clipping, which is NOT the gradient clipping acting on individual weights, but clipping on the probability ratio, not the whole probability distribution.