GPT-2 is a GPT model introduced in 2019. Its largest version has approximately 1.5 billion parameters.
Architecture and training
- Uses a decoder-only Transformer
- Predicts text from left to right
- Uses WebText as its main reported training dataset
Changes from GPT-1
- Increases the model size from about 117 million to 1.5 billion parameters
- Shows In-Context Learning behaviour for some tasks
- Uses byte-level BPE for tokenization
Byte-level BPE starts from 256 possible byte values. It can represent punctuation and other characters through their encoded bytes.
Task behaviour
The prompt can give a task context that guides the continuation. This behaviour became stronger in GPT-3, which demonstrated zero-shot and few-shot performance across more tasks.