Reinforcement Learning from Human Feedback (RLHF)

Reinforcement Learning from Human Feedback (RLHF) uses human judgements to guide a model towards preferred behaviour.

Human-feedback process

1. Give a prompt to the language model
2. Generate several candidate answers
3. Ask human labelers to rank the answers
4. Train a reward model to predict the human preferences
5. Use the reward model's scores to update the language model through reinforcement learning

candidate answers -> human rankings -> reward model
language model answers -> reward scores -> language model updates

Human rankings provide comparison data. The reward model then supplies scores during reinforcement learning. InstructGPT method

Demonstrations and rankings

Signal	What humans provide	How it is used
Supervised demonstration	A desired answer	Train the model to imitate the answer
Preference ranking	An ordering of candidate answers	Train the reward model
Reward score	Produced by the trained reward model	Guide reinforcement-learning updates

Connection to language models

InstructGPT and the original ChatGPT use demonstrations followed by human-feedback training. These stages update the model weights.

In-Context Learning supplies task information in the prompt while the model weights stay fixed. See LLM Training Stages for the complete sequence.