Overview
- a policy gradient method makes high-reward outputs more likely, even when the reward cannot be differentiated
- goal: maximise the expected reward of the answers the model samples
- problem: the reward comes after a sampling step, so no gradient can flow back from the reward itself
- fix: backpropagate through the log-probability of the answer instead, and use the reward only as a weight on that gradient
- REINFORCE samples a few answers and pushes each one up or down by its reward, so the reward decides what "better" means
A policy gradient method trains a model by directly increasing the probability of outputs that get a high reward. REINFORCE is the classic policy gradient algorithm. It lets a language model learn from a score such as human preference, even though that score cannot be differentiated.
Optimising a reward instead of a loss
Suppose a language model writes summaries, and for each sampled summary there is a way to get a human reward , a real number where higher is better.
Example source article: "A magnitude 4.2 earthquake shook the San Francisco area Friday at 4:42 a.m. PT, strong enough to overturn unstable objects."
- summary : "An earthquake hit San Francisco. There was minor property damage, but no injuries." gets
- summary : "The Bay Area has good weather but is prone to earthquakes and wildfires." gets
The goal is to maximise the expected reward of samples from the model:
where:
- is the objective, the average reward the model earns
- is the set of model parameters
- is the prompt
- is an answer sampled from the model
- is the probability that the model gives to answer for prompt
- is the score that the answer receives after it is generated
Example prompt: explain why a metal spoon feels cold.
- a clear, correct explanation has and
- a vague explanation has and
- a confident error has and
Training must move probability away from the confident error and towards the clear explanation.
Why reinforcement learning, and why now
Reinforcement learning (RL) matured in games before it became practical for language models.
- 1992, policy gradients: REINFORCE (Williams) gave a way to optimise expected reward
- 2013, deep RL breakthrough: neural policies achieved landmark game-playing results (Mnih et al.)
- 2017, stable large-model updates: PPO made policy optimisation more robust and practical (Schulman et al.)
- 2019 to 2022, RL reaches language models: human feedback guided summaries and instruction following (Ziegler, Stiennon, Ouyang)
Adoption lagged because RL with large neural models was unstable and hard to tune. PPO and scalable human-feedback pipelines reduced these barriers. The key change was not a new objective. It was having algorithms and feedback systems that could optimise the objective at scale.
Why RL on large models was unstable:
- noisy gradient: the gradient comes from a few sampled answers with one score each, so two batches can point in quite different directions
- credit assignment: one score arrives at the end of hundreds of tokens, so it is hard to tell which tokens earned it
- self-made data: the model trains on its own answers, so one bad update gives worse answers, and the model then learns from those worse answers, which can make training collapse
- step size: a small change in the weights can cause a large change in the output probabilities, so the learning rate is hard to set
- expensive tuning: each run costs a lot of compute and there are many hyperparameters, so few settings can be tried
- vanishing gradients are not the main cause, because residual connections and layer norm already handle them in Transformers
Problem: how to move the parameters
The natural idea is gradient ascent:
where:
- is the parameter value at step
- is the learning rate
- is the gradient with respect to the parameters
This raises two questions:
- how can the expectation be estimated, when there are too many possible answers to list
- what if the reward function is not differentiable
Why the reward cannot be backpropagated
Backpropagation is still used to update the weights. What does not work is backpropagating from the reward, because the score arrives only after a discrete text sample.
- generate: the model chooses tokens and produces one answer
- judge: a person or a reward model gives a score
- learn: the parameters change so that useful answers happen again
To backpropagate from the reward, a gradient would have to pass along every arrow of this path:
θ -> logits -> sample a token -> text -> judge -> RTwo arrows have no gradient:
- sampling a token is a discrete choice, so a small change in the logits does not smoothly change which token is picked
- the judge may be a person, and a person's score has no gradient
So the gradient of the reward, , is not available. A shorter path does work:
θ -> logits -> log-probability of the chosen tokensThis is a normal network output, so the gradient of the answer's log-probability, , is available. The method backpropagates through the log-probability of the answer, and the reward only scales that gradient.
Log-probability trick
The identity is:
Insert it into the objective:
where:
- is a sum over every possible answer, which is impossible to compute
- is an average over answers that the model actually samples
So the impossible sum becomes an average over answers the model can actually sample.
REINFORCE estimator
REINFORCE does not list every answer. It samples answers and averages their signals:
and then updates the parameters:
where:
- is the estimated gradient, or signal
- is the number of sampled answers in the batch
- is the -th sampled answer for prompt
- is the reward of that answer
- is the direction that makes answer more likely
Example rewards in one batch: a useful checklist (), a relevant but vague answer (), an invented fact (), an honest limitation (). Only the sampled answers receive direct probability pressure in this batch.
How the reward enters the loss
The reward never goes into the network. It is a number from outside the model that multiplies each answer's loss term.
One training step on a batch of answers. Steps 1 to 3 are done for each answer (each sample) on its own:
- generate: the model writes answer one token at a time, drawing each token at random from its predicted probabilities; the log-probability of each chosen token is recorded, and their sum is
- score: a person or a reward model gives one number for the whole answer
- build the loss term:
- average: the batch loss is
- backpropagate and update: the gradient is , which is , so each only scales its own answer's gradient, and one step is taken for the whole batch
Example of step 1, for the prompt "Metal feels cold because":
- "heat" is drawn with probability 0.3, so
- "flows" is drawn with probability 0.7, so
- "away" is drawn with probability 0.8, so
- the answer's log-probability is , which equals
What controls:
- sign: pushes the answer's probability up, and pushes it down
- size: a larger gives a larger push
The minus sign turns gradient ascent on the reward into gradient descent on a loss, which is what standard optimisers do. In PyTorch:
loss = -(reward.detach() * log_prob).mean()
loss.backward()
optimizer.step()detach() keeps the reward a constant, so no gradient tries to flow into it.
Link to supervised fine-tuning:
- SFT loss: , which is cross-entropy on an answer written by a human
- REINFORCE loss: , which is the same cross-entropy on the model's own answer, scaled by its reward
- so REINFORCE is SFT on the model's own samples, weighted by reward
- best-of-N sampling is the simplest case, with for the top answer and for the rest
Reward controls the direction
Prompt: a traveller asks how to rebook a cancelled flight.
- positive signal: an answer that gives concrete options and admits uncertainty gets , so its probability goes up
- negative signal: an answer that invents a phone number and promises a refund gets , so its probability goes down
- gives no direct push for that sample
If the invented answer wrongly received a positive score, REINFORCE would reinforce it. So the reward defines what "better" means.
Note
A raw reward can mislead the update, because the same score can be good on one prompt and poor on another. The fix is the Advantage Function.