Overview
- InstructGPT is GPT-3 trained to follow instructions with RLHF
- three stages: SFT on human-written answers, a reward model from rankings, then RL
- the prompts came from labellers in three ways: plain, few-shot and user-based
- results show GPT-3 just continuing the pattern, while InstructGPT actually answers
InstructGPT is a set of GPT-3 models fine-tuned to follow instructions. The training method combines human demonstrations with human preferences about generated answers.
Training stages
- Supervised fine-tuning: train on prompts paired with answers written by human labelers
- Reward model training: generate several answers, collect human rankings, and train a model to predict those preferences
- Reinforcement learning: update the language model to produce answers that receive higher reward scores
The human-ranking and reinforcement-learning process is RLHF.
pre-trained GPT-3 -> demonstration fine-tuning
-> human rankings -> reward model
-> reinforcement learning -> InstructGPTScale
InstructGPT and ChatGPT scaled RLHF to tens of thousands of tasks. The prompt dataset holds about 30k tasks.
The InstructGPT paper splits its prompts by stage: about 13k prompts for SFT, 33k for the reward model and 31k for PPO. So "about 30k" is a round figure for the size of the prompt pool, not the SFT set alone.
Tasks written by labellers
The prompts were collected from labellers in three ways:
- plain: labellers make up an arbitrary task, while making sure the tasks are diverse
- few-shot: labellers make up an instruction plus several query and response pairs for it
- user-based: labellers write prompts that match use-cases stated in waitlist applications to the OpenAI API
Example use-cases and prompts:
- brainstorming: "List five ideas for how to regain enthusiasm for my career"
- generation: "Write a short story where a bear goes to the beach, makes friends with a seal, and then returns home"
Purpose
A base language model learns to continue text. InstructGPT training teaches it to respond to the instruction in that text. The training process updates model weights. InstructGPT training method
Results
Prompt: "Explain the moon landing to a 6 year old in a few sentences."
- GPT-3 continues the pattern with more instructions: "Explain the theory of gravity to a 6 year old. Explain the theory of relativity to a 6 year old in a few sentences. Explain the big bang theory to a 6 year old. Explain evolution to a 6 year old."
- InstructGPT answers the instruction: "People went to the moon, and they took pictures of what they saw, and sent them back to the earth so we could all see them."
Prompt: "Write a short poem about a wise frog."
- GPT-3 again writes more prompts, such as "Write a short story in which a character has two different names."
- InstructGPT writes the poem: "The frog is a symbol of wisdom / He knows all the secrets of the world / He is a master of disguise / And a great teacher of life ..."
Connection to ChatGPT
The original ChatGPT uses a related method with dialogue data and a GPT-3.5 base model. See LLM Training Stages for the wider training process.