The vanishing gradient problem occurs in very deep networks during backpropagation, where gradients shrink exponentially as they travel back through layers which eventually becoming so small that early layers stop learning.

Problem

During loss.backward(), gradients are calculated and sent backwards through every layer. In a deep network:

  1. The gradient reaching the earlier hidden layers having been multiplied many times
  2. Each multiplication can shrink the value toward zero (because its chain rule)
  3. The gradient effectively "vanishes" before it reaches the early layers

Consequences

SymptomEffect
Early layers stop learningWeights near the input never update
Only the last few layers updateThe network learns very shallow features
Deep networks underperform shallow onesDepth provides no benefit if gradients can't flow

Connection to RNNs and LSTMs

A recurrent neural network (RNN) reuses its hidden state across time steps. During backpropagation through time, gradients pass through this sequence of repeated operations.

  • Long sequences require gradients to travel through many time steps
  • Repeated small factors can make those gradients very small
  • Earlier words then receive a weak learning signal, so the model struggles with long-term dependencies

LSTM was designed to reduce this problem. Its cell state carries information over time. The forget, input, and output gates control which information is retained, added, and exposed.

The cell-state path helps gradients and useful information pass across longer sequences.

Common Fixes

  • Residual connections (ResNet) which adds skip connections so gradients have a direct path backward
  • Better activations such as ReLU avoids the saturation that crushes sigmoid gradients
  • Batch normalisation keeps activations in a healthy range at each layer

Exploding gradients

Gradient Clipping caps gradients that become too large. LSTM memory paths and residual connections address the separate problem of weak gradient flow.