Overview

  • a reward model is a network that learns to score answers the way humans would
  • humans give pairwise choices (A or B), because these are easier than giving a number
  • it is trained with the Bradley-Terry loss, which pushes the winner's score above the loser's
  • it is checked against held-out human choices, and it is the second stage of RLHF

A reward model is a neural network that learns to predict human judgement. It takes a prompt and an answer and outputs a single number, the scalar reward. It replaces a human during RL training, because humans are expensive and inconsistent.

Why not ask humans every time

  • direct human feedback: a person must judge every new answer, which is slow and costly
  • learned reward model: people label a smaller set of comparisons once, and the model then scores many new answers

Why pairwise choices are easier

Example: a passenger misses the last train home. Which answer is more useful?

  • answer A: "Try another route."
  • answer B: "Check the night bus, compare a taxi, and message anyone waiting for you."

Choosing between A and B is easy. Inventing a calibrated score, such as 7.3 out of 10, is hard and inconsistent between people. So human data is collected as pairwise comparisons.

How the reward model learns

  • human comparison: answer B is preferred to answer A, written
  • what the model should learn:
  • Bradley-Terry probability: the chance that B beats A depends only on the difference of the two scores

where:

  • is the reward model with parameters
  • is the score it gives to answer B
  • is the sigmoid function, which turns any number into a probability between 0 and 1

Many choices teach one scoring function. Humans do not need to give a score to every future answer.

Training setup

  • the input is a pair: the prompt with the selected (preferred) completion, and the same prompt with the rejected completion
  • a Transformer reads each one and outputs a scalar reward
  • the loss pushes the model to increase the difference between the reward of the selected and the rejected completion

Bradley-Terry loss

Human raters give a prompt and a pair of responses, the winner and the loser . The reward model minimises the expected value of:

with respect to the reward parameters , where:

  • is the prompt
  • is the winning (preferred) response
  • is the losing (rejected) response
  • is the reward the model gives to response
  • the loss is small when the winner's reward is clearly above the loser's reward
  • both rewards come from the same reward model

This is the loss used by InstructGPT. If the reward is linear in its input variables, it is the same as a maximum-likelihood fit of a binomial preference function.

In plain words, each comparison is a yes or no outcome: did the winner win? The loss is logistic regression on the score difference. It fits the reward model so that the observed choices become as likely as possible. "Binomial" only means there are two possible outcomes for each pair.

Check the reward model before using it

The reward model is evaluated on how well it predicts held-out human judgements.

  • validation accuracy rises with model size and with the amount of comparison data (8k to 64k)
  • a large enough reward model trained on enough data approaches the performance of a single human, though it stays below an ensemble of humans

Note

The reward model is the second of the three stages in RLHF.