Overview

  • InstructGPT is GPT-3 trained to follow instructions with RLHF
  • three stages: SFT on human-written answers, a reward model from rankings, then RL
  • the prompts came from labellers in three ways: plain, few-shot and user-based
  • results show GPT-3 just continuing the pattern, while InstructGPT actually answers

InstructGPT is a set of GPT-3 models fine-tuned to follow instructions. The training method combines human demonstrations with human preferences about generated answers.

Training stages

  1. Supervised fine-tuning: train on prompts paired with answers written by human labelers
  2. Reward model training: generate several answers, collect human rankings, and train a model to predict those preferences
  3. Reinforcement learning: update the language model to produce answers that receive higher reward scores

The human-ranking and reinforcement-learning process is RLHF.

pre-trained GPT-3 -> demonstration fine-tuning
                 -> human rankings -> reward model
                 -> reinforcement learning -> InstructGPT

Scale

InstructGPT and ChatGPT scaled RLHF to tens of thousands of tasks. The prompt dataset holds about 30k tasks.

The InstructGPT paper splits its prompts by stage: about 13k prompts for SFT, 33k for the reward model and 31k for PPO. So "about 30k" is a round figure for the size of the prompt pool, not the SFT set alone.

Tasks written by labellers

The prompts were collected from labellers in three ways:

  • plain: labellers make up an arbitrary task, while making sure the tasks are diverse
  • few-shot: labellers make up an instruction plus several query and response pairs for it
  • user-based: labellers write prompts that match use-cases stated in waitlist applications to the OpenAI API

Example use-cases and prompts:

  • brainstorming: "List five ideas for how to regain enthusiasm for my career"
  • generation: "Write a short story where a bear goes to the beach, makes friends with a seal, and then returns home"

Purpose

A base language model learns to continue text. InstructGPT training teaches it to respond to the instruction in that text. The training process updates model weights. InstructGPT training method

Results

Prompt: "Explain the moon landing to a 6 year old in a few sentences."

  • GPT-3 continues the pattern with more instructions: "Explain the theory of gravity to a 6 year old. Explain the theory of relativity to a 6 year old in a few sentences. Explain the big bang theory to a 6 year old. Explain evolution to a 6 year old."
  • InstructGPT answers the instruction: "People went to the moon, and they took pictures of what they saw, and sent them back to the earth so we could all see them."

Prompt: "Write a short poem about a wise frog."

  • GPT-3 again writes more prompts, such as "Write a short story in which a character has two different names."
  • InstructGPT writes the poem: "The frog is a symbol of wisdom / He knows all the secrets of the world / He is a master of disguise / And a great teacher of life ..."

Connection to ChatGPT

The original ChatGPT uses a related method with dialogue data and a GPT-3.5 base model. See LLM Training Stages for the wider training process.