Overview
- RLHF uses human judgements to steer a model towards answers people prefer
- three phases: supervised fine-tuning, reward model training, then RL with PPO
- the RL reward is the reward-model score minus a KL penalty, so the model stays close to a frozen reference
- PPO in RLHF uses four models (policy, reference, reward and value), and the note ends with gains over SFT
Reinforcement Learning from Human Feedback (RLHF) uses human judgements to guide a model towards preferred behaviour.
Human-feedback process
- Give a prompt to the language model
- Generate several candidate answers
- Ask human labelers to rank the answers
- Train a reward model to predict the human preferences
- Use the reward model's scores to update the language model through reinforcement learning
candidate answers -> human rankings -> reward model
language model answers -> reward scores -> language model updatesHuman rankings provide comparison data. The reward model then supplies scores during reinforcement learning. InstructGPT method
Demonstrations and rankings
| Signal | What humans provide | How it is used |
|---|---|---|
| Supervised demonstration | A desired answer | Train the model to imitate the answer |
| Preference ranking | An ordering of candidate answers | Train the reward model |
| Reward score | Produced by the trained reward model | Guide reinforcement-learning updates |
Training phases
RLHF is built in three phases: pretrain and fine-tune, fit a reward model, then optimise against it under a KL leash.

- base model: pretraining plus supervised fine-tuning gives a model that is already instruction-following, helpful and chatty
- preference collection and training: humans compare answers, and a Reward Model is fitted to their choices
- RL optimisation: the language model is improved by reinforcement learning against the reward model
The full pipeline, as used for InstructGPT:

- step 1: collect demonstration data and train a supervised policy; a prompt is sampled, a labeller writes the desired output, and the data fine-tunes GPT-3 with supervised learning (this is instruction tuning)
- step 2: collect comparison data and train a reward model; several model outputs are sampled for one prompt, a labeller ranks them from best to worst, and the ranking trains the reward model
- step 3: optimise a policy against the reward model with PPO; a new prompt is sampled, the policy writes an output, the reward model gives it a reward, and the reward updates the policy
RL notation inside the training loop

- the agent is the language model being trained, written as the policy
- the environment is the reward model and the other infrastructure
- is the state, which is the next prompt (and the text so far)
- is the action, which is the completion to the prompt, chosen as
- is the reward returned by the environment
Putting it all together
What is already available before RL:
- a supervised language model
- a reward model trained from pairwise choices
- a frozen reference copy of the language model
What PPO then does:
- generate an answer
- score the answer
- increase its probability when the signal is positive
- limit each update and penalise drift
The training signal is the reward-model score minus a drift penalty:
where:
- is the final reward used for RL on answer
- is the score from the reward model with parameters
- is the probability of under the model being trained
- is the probability of under the frozen reference model (the model before RLHF)
- measures how far the model has drifted from the reference, and its average is the KL divergence
- is a weight that sets how strongly drift is penalised
The reference model keeps the improved answers close to the language that was learned before RLHF.
The symbols and mean the same thing, the probability that the model gives to an answer. RL papers usually write for the policy.
The four parts map to four roles:
- policy: drafts two answers
- human preference: chooses the stronger one
- reward model: learns a scoring rule from those choices
- PPO: revises the policy in small steps
PPO for RLHF: four models, one update

One PPO update in RLHF uses four models:
- policy model: the language model being trained, which writes the answer
- reference model: frozen, used for the KL penalty
- reward model: frozen, gives the score
- value model: trained, estimates the expected reward ; the rewards and values are combined by GAE (generalised advantage estimation) into the advantage
The policy is trained to maximise the advantage:
where:
- is a prompt from the prompt dataset
- is an answer sampled from the old policy
- is the number of tokens in the answer
- is the -th token and is all tokens before it
- the fraction is the probability ratio of PPO for each token
- is the advantage
This short form leaves out the clipping. The full objective applies the and clip of PPO to each token's ratio, exactly as in the clipped objective.
The reward uses a penalty per token for straying from the reference policy:
where:
- is the reward at token step
- is the reward-model score
- is the supervised fine-tuned model used as the reference
- is the penalty weight
In practice the reward model reads the whole answer, so its score is given once, at the last token. The KL penalty is given at every token. So most tokens receive only the KL part, and the final token also receives the reward-model score. The value model then spreads this end reward back over the earlier tokens when the advantage is computed.
The value function, and so the advantage, must be estimated, which is why a separate value model is needed.
Gains over pretraining and fine-tuning

On summarisation, the fraction of model summaries that humans prefer over reference summaries is plotted against model size (1.3B to 12.9B):
- pretrain only () is lowest
- supervised learning () is better but stays below the human reference summaries
- human feedback () is the only method above the reference line, and it is better than supervised learning at every model size
Connection to language models
InstructGPT and the original ChatGPT use demonstrations followed by human-feedback training. These stages update the model weights.
In-Context Learning supplies task information in the prompt while the model weights stay fixed. See LLM Training Stages for the complete sequence.