Gradient Clipping

Gradient clipping limits large gradients before Optimisers use them to update a model. It can improve training stability when gradients grow too large, especially in recurrent networks.

Exploding gradients are excessively large gradients. Vanishing gradients are gradients that become too small. Clipping reduces large values; small gradients require a separate solution.

Two methods

Method	Rule	Effect
By value	Limit each component to the range $[-c,c]$	Can change the gradient's direction
By norm	Scale the vector down if its length exceeds $c$	Preserves the direction of that vector

Here, $c$ is a positive threshold. For the gradient $g=[3,4]$:

• Its Euclidean norm is $\sqrt{3^2+4^2}=5$
• Value clipping with $c=1$ gives $[1,1]$
• Norm clipping with $c=1$ multiplies both components by $1/5$, giving $[0.6,0.8]$
• Norm clipping retains a gradient whose norm already meets the limit

Global norm clipping treats all parameter gradients as one vector. Per-tensor norm clipping limits each parameter tensor separately. These produce different results when the model has several parameter tensors.

Code

PyTorch

This small loss makes the gradient exactly [3, 4] before clipping:

import torch
 
w = torch.nn.Parameter(torch.tensor([1., 1.]))
optimizer = torch.optim.SGD([w], lr=0.1)
 
optimizer.zero_grad()
loss = 3 * w[0] + 4 * w[1]
loss.backward()
torch.nn.utils.clip_grad_value_([w], clip_value=0.5)
print(w.grad.tolist())
optimizer.step()

[0.5, 0.5]

Read the order:

• backward() creates w.grad with values [3, 4]
• clip_grad_value_() changes those stored gradients to [0.5, 0.5]
• The final underscore indicates an operation that changes existing values
• step() uses the clipped values, so both parameters decrease from 1.0 to about 0.95

For global norm clipping, replace the clipping line before running the example:

torch.nn.utils.clip_grad_norm_([w], max_norm=1.0)

This gives gradients close to [0.6, 0.8]. In a full model, pass model.parameters() in place of [w]. The PyTorch norm function combines all supplied parameter gradients.

TensorFlow and Keras

import tensorflow as tf
 
w = tf.Variable([1., 1.])
optimizer = tf.keras.optimizers.SGD(
    learning_rate=0.1, clipvalue=0.5
)
with tf.GradientTape() as tape:
    loss = 3 * w[0] + 4 * w[1]
gradient = tape.gradient(loss, w)
optimizer.apply_gradients([(gradient, w)])
print(w.numpy().round(2).tolist())

The result is approximately [0.95, 0.95].

Read the data flow:

• GradientTape records operations used to calculate loss
• tape.gradient() produces the gradient tensor [3, 4]
• (gradient, w) pairs the gradient with the variable it updates
• apply_gradients() applies the configured clipping before the update

For a model trained with compile() and fit(), put the same setting in its optimiser, such as tf.keras.optimizers.Adam(clipvalue=0.5).

Choose one clipping setting:

• clipvalue=0.5 limits each gradient component
• clipnorm=1.0 limits each parameter tensor's gradient norm separately
• global_clipnorm=1.0 limits the combined norm across all parameter tensors

These names and scopes follow the Keras optimiser API.