The vanishing gradient problem occurs in very deep networks during backpropagation, where gradients shrink exponentially as they travel back through layers which eventually becoming so small that early layers stop learning.
Problem
During loss.backward(), gradients are calculated and sent backwards through every layer. In a deep network:
- The gradient reaching the earlier hidden layers having been multiplied many times
- Each multiplication can shrink the value toward zero (because its chain rule)
- The gradient effectively "vanishes" before it reaches the early layers
Consequences
| Symptom | Effect |
|---|---|
| Early layers stop learning | Weights near the input never update |
| Only the last few layers update | The network learns very shallow features |
| Deep networks underperform shallow ones | Depth provides no benefit if gradients can't flow |
Connection to RNNs and LSTMs
A recurrent neural network (RNN) reuses its hidden state across time steps. During backpropagation through time, gradients pass through this sequence of repeated operations.
- Long sequences require gradients to travel through many time steps
- Repeated small factors can make those gradients very small
- Earlier words then receive a weak learning signal, so the model struggles with long-term dependencies
LSTM was designed to reduce this problem. Its cell state carries information over time. The forget, input, and output gates control which information is retained, added, and exposed.
The cell-state path helps gradients and useful information pass across longer sequences.
Common Fixes
- Residual connections (ResNet) which adds skip connections so gradients have a direct path backward
- Better activations such as ReLU avoids the saturation that crushes sigmoid gradients
- Batch normalisation keeps activations in a healthy range at each layer
Exploding gradients
Gradient Clipping caps gradients that become too large. LSTM memory paths and residual connections address the separate problem of weak gradient flow.