Mixed precision training runs most of a network's maths in 16-bit floats and keeps the sensitive parts in 32-bit. It roughly halves activation memory and speeds up training on GPUs with tensor cores, usually with no loss in accuracy. In PyTorch it is called AMP (automatic mixed precision).
Formats
| Format | Bits | Range | Note |
|---|---|---|---|
| float32 | 32 | about | the safe default |
| float16 | 16 | about to | small range, needs loss scaling |
| bfloat16 | 16 | same range as float32 | fewer precision bits, usually no loss scaling needed |
What Runs In Which Precision
- matrix multiplies and convolutions run in 16-bit, which is where the speed comes from
- reductions such as softmax, losses and normalisation statistics stay in 32-bit
- a 32-bit master copy of the weights is updated by the optimiser
Loss Scaling
- many gradients are smaller than float16 can represent, so they would round to zero (underflow)
- multiply the loss by a large factor before backpropagation, so gradients are times larger
- divide the gradients by before the optimiser step
- a dynamic scaler raises while training is stable, and lowers it when it finds infinities or NaNs, skipping that step
In PyTorch
scaler = torch.amp.GradScaler("cuda")
with torch.amp.autocast("cuda"):
logits = model(x)
loss = criterion(logits, y)
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()Things To Watch
- very large logits can overflow in float16, so clamping logits or using bfloat16 helps
- data preprocessing that does not need gradients can run under
torch.no_grad()to save memory