Mixed precision training runs most of a network's maths in 16-bit floats and keeps the sensitive parts in 32-bit. It roughly halves activation memory and speeds up training on GPUs with tensor cores, usually with no loss in accuracy. In PyTorch it is called AMP (automatic mixed precision).

Formats

FormatBitsRangeNote
float3232about the safe default
float1616about to small range, needs loss scaling
bfloat1616same range as float32fewer precision bits, usually no loss scaling needed

What Runs In Which Precision

  • matrix multiplies and convolutions run in 16-bit, which is where the speed comes from
  • reductions such as softmax, losses and normalisation statistics stay in 32-bit
  • a 32-bit master copy of the weights is updated by the optimiser

Loss Scaling

  • many gradients are smaller than float16 can represent, so they would round to zero (underflow)
  • multiply the loss by a large factor before backpropagation, so gradients are times larger
  • divide the gradients by before the optimiser step
  • a dynamic scaler raises while training is stable, and lowers it when it finds infinities or NaNs, skipping that step

In PyTorch

scaler = torch.amp.GradScaler("cuda")
with torch.amp.autocast("cuda"):
    logits = model(x)
    loss = criterion(logits, y)
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()

Things To Watch

  • very large logits can overflow in float16, so clamping logits or using bfloat16 helps
  • data preprocessing that does not need gradients can run under torch.no_grad() to save memory