RLHF (reinforcement learning from human feedback, Ouyang et al. 2022) trains the model on what people prefer. Scoring one answer from 1 to 10 is hard, but picking the better of 2 answers is easy, like a taste test.
3 steps
- people pick the better of 2 answers
- a reward model learns to predict their pick
- the LLM is trained to score highly under the reward model, on a leash
Remembering tip
A coach (the reward model) learns what the judges like, then trains the athlete (the LLM). The leash stops the athlete from gaming the blind spots of the coach.
Formulas
| Piece | Meaning |
|---|---|
| The symbol reads "is preferred to", not "greater than": people preferred answer (winner) to answer (loser) | |
| and | The reward model score, and the sigmoid that turns a score gap into a probability |
| The leash: a penalty for drifting away from the SFT model |
Example: scores of and give , so the reward model says the winner is preferred with probability .
DPO
In practice many teams use DPO (direct preference optimisation, Rafailov et al. 2023). It turns the 2 lines above into one simple loss on the preference pairs, with no separate reward model.
Code
import numpy as np
def sigmoid(x):
return 1 / (1 + np.exp(-x))
r_w, r_l = 2.0, 0.5 # reward model scores of the winner and the loser
print(round(sigmoid(r_w - r_l), 2)) # 0.82
loss = -np.log(sigmoid(r_w - r_l)) # reward model loss for this pair
print(round(loss, 3)) # 0.201Notes:
- the loss is small when the reward model already scores the winner higher than the loser
- a larger gap gives a larger probability and a smaller loss
When a program can check the answer, no human is needed, see Reinforcement Learning with Verifiable Rewards (RLVR).