InstructGPT is a set of GPT-3 models fine-tuned to follow instructions. The training method combines human demonstrations with human preferences about generated answers.
Training stages
- Supervised fine-tuning: train on prompts paired with answers written by human labelers
- Reward model training: generate several answers, collect human rankings, and train a model to predict those preferences
- Reinforcement learning: update the language model to produce answers that receive higher reward scores
The human-ranking and reinforcement-learning process is RLHF.
pre-trained GPT-3 -> demonstration fine-tuning
-> human rankings -> reward model
-> reinforcement learning -> InstructGPTPurpose
A base language model learns to continue text. InstructGPT training teaches it to respond to the instruction in that text. The training process updates model weights. InstructGPT training method
Connection to ChatGPT
The original ChatGPT uses a related method with dialogue data and a GPT-3.5 base model. See LLM Training Stages for the wider training process.