GPT-1

GPT-1 is the first GPT model. It was introduced in 2018 with approximately 117 million parameters.

Training and architecture

• Uses Transformer decoder blocks
• Learns left-to-right next-token prediction from unlabelled text
• Uses BooksCorpus for pre-training
• Uses learned absolute position embeddings

Downstream tasks

GPT-1 relied heavily on fine-tuning for individual downstream tasks.

unlabelled text -> pre-trained GPT-1
               -> labelled task data -> task-specific fine-tuning

Fine-tuning updates the model weights for the selected task. Later models, including GPT-2 and GPT-3, showed stronger task behaviour from instructions or examples in a prompt.