Label smoothing replaces hard targets such as 1 and 0 with slightly softer ones, so a model is not pushed to be completely certain. It acts as a regulariser on the Loss Functions used for classification.
Formula
For classes and smoothing :
Binary case, as often written for one sigmoid output:
- with , targets 1 and 0 become 0.95 and 0.05
Why It Helps
- with hard targets, cross-entropy keeps falling as a logit grows, so the model pushes logits towards infinity
- very large logits mean over-confident predictions and poor calibration (a predicted 99% is not right 99% of the time)
- soft targets give the loss a minimum at a finite logit, so the model stops growing them
- it also reduces the damage from noisy labels, because a wrong label costs less
Costs
- probabilities are slightly less sharp, so a threshold tuned without smoothing may need adjusting
- too much smoothing hides real confidence and can lower accuracy
Related Tricks
- logit clamping (for example to ) puts a hard limit on logits, which protects the loss from overflow, especially in mixed precision
- in PyTorch,
CrossEntropyLoss(label_smoothing=0.1)does it for multi-class, and forBCEWithLogitsLossyou smooth the target tensor yourself