Scaling laws describe how a language model improves as the model size, the amount of training data and the compute budget grow.
Single Important Point
- for a fixed compute budget, the model size and the number of training tokens must be balanced
- a very large model trained on too few tokens wastes compute
- a small model trained on a huge number of tokens also wastes compute
What it rules out
- always maximising model size is wrong
- always maximising data is wrong
- the learning rate is not the lever here
The models these laws were measured on are in Generative Pre-trained Transformer (GPT).