RLHF (reinforcement learning from human feedback, Ouyang et al. 2022) trains the model on what people prefer. Scoring one answer from 1 to 10 is hard, but picking the better of 2 answers is easy, like a taste test.

3 steps

  • people pick the better of 2 answers
  • a reward model learns to predict their pick
  • the LLM is trained to score highly under the reward model, on a leash

Remembering tip

A coach (the reward model) learns what the judges like, then trains the athlete (the LLM). The leash stops the athlete from gaming the blind spots of the coach.

Formulas

PieceMeaning
The symbol reads "is preferred to", not "greater than": people preferred answer (winner) to answer (loser)
and The reward model score, and the sigmoid that turns a score gap into a probability
The leash: a penalty for drifting away from the SFT model

Example: scores of and give , so the reward model says the winner is preferred with probability .

DPO

In practice many teams use DPO (direct preference optimisation, Rafailov et al. 2023). It turns the 2 lines above into one simple loss on the preference pairs, with no separate reward model.

Code

import numpy as np
 
def sigmoid(x):
    return 1 / (1 + np.exp(-x))
 
r_w, r_l = 2.0, 0.5                      # reward model scores of the winner and the loser
print(round(sigmoid(r_w - r_l), 2))      # 0.82
 
loss = -np.log(sigmoid(r_w - r_l))       # reward model loss for this pair
print(round(loss, 3))                    # 0.201

Notes:

  • the loss is small when the reward model already scores the winner higher than the loser
  • a larger gap gives a larger probability and a smaller loss

When a program can check the answer, no human is needed, see Reinforcement Learning with Verifiable Rewards (RLVR).