Label smoothing replaces hard targets such as 1 and 0 with slightly softer ones, so a model is not pushed to be completely certain. It acts as a regulariser on the Loss Functions used for classification.

Formula

For classes and smoothing :

Binary case, as often written for one sigmoid output:

  • with , targets 1 and 0 become 0.95 and 0.05

Why It Helps

  • with hard targets, cross-entropy keeps falling as a logit grows, so the model pushes logits towards infinity
  • very large logits mean over-confident predictions and poor calibration (a predicted 99% is not right 99% of the time)
  • soft targets give the loss a minimum at a finite logit, so the model stops growing them
  • it also reduces the damage from noisy labels, because a wrong label costs less

Costs

  • probabilities are slightly less sharp, so a threshold tuned without smoothing may need adjusting
  • too much smoothing hides real confidence and can lower accuracy
  • logit clamping (for example to ) puts a hard limit on logits, which protects the loss from overflow, especially in mixed precision
  • in PyTorch, CrossEntropyLoss(label_smoothing=0.1) does it for multi-class, and for BCEWithLogitsLoss you smooth the target tensor yourself