GPT-3 is a GPT model introduced in 2020. Its largest version has 175 billion parameters.
Architecture and training
- Uses a decoder-only Transformer
- Learns left-to-right next-token prediction
- Trains on a large text mixture that includes filtered Common Crawl data
In-context learning
GPT-3 showed strong In-Context Learning:
| Setting | Information in the prompt | Weight update during the task |
|---|---|---|
| Zero-shot | Task description | No |
| One-shot | Task description and one example | No |
| Few-shot | Task description and several examples | No |
A prompt can demonstrate patterns such as simple arithmetic, spelling correction, or translation. The model uses those patterns to produce the next answer.
For some tasks, zero-shot or few-shot performance is comparable to task-specific fine-tuned models. This result depends on the task.
Connection to instruction following
InstructGPT fine-tunes GPT-3 using human demonstrations and RLHF. ChatGPT extends this type of training to dialogue.