JJ

Gradient Descent

2 min read

Gradient descent updates parameters to reduce a loss. A gradient describes how the loss changes when a parameter changes.

For each update:

  • Calculate the prediction and its loss
  • Calculate the gradient of the loss with respect to each parameter
  • Move each parameter opposite to its gradient, using the selected learning rate

A positive gradient makes the update decrease the parameter. A negative gradient makes it increase the parameter. The Learning Rate controls the step size.

For a parameter of 2.0, a gradient of 0.4, and a learning rate of 0.1, subtract 0.04. The new parameter is 1.96.

Full batches and small batches

Full-batch gradient descent calculates each update from the complete training set. Mini-batch training uses a smaller group of samples per update. An optimiser named SGD can use either batch arrangement in code.

Code

PyTorch: one parameter update

import torch
 
w = torch.nn.Parameter(torch.tensor(2.0))
optimizer = torch.optim.SGD([w], lr=0.1)
optimizer.zero_grad()
loss = 0.4 * w
loss.backward()
print(round(w.grad.item(), 2))  # 0.4
optimizer.step()
print(round(w.item(), 2))       # 1.96

Read the sequence:

  • Parameter makes w a trainable tensor
  • [w] supplies this parameter to the optimiser
  • zero_grad() clears earlier gradients
  • backward() calculates the gradient; it leaves the parameter value unchanged
  • step() changes the parameter using that gradient

The simple loss isolates one update. A full model uses a prediction loss based on its target values.

TensorFlow/Keras: the same update

import tensorflow as tf
 
w = tf.Variable(2.0)
optimizer = tf.keras.optimizers.SGD(learning_rate=0.1)
with tf.GradientTape() as tape:
    loss = 0.4 * w
gradient = tape.gradient(loss, w)
optimizer.apply_gradients([(gradient, w)])
print(round(float(w.numpy()), 2))  # 1.96

GradientTape records the calculation. tape.gradient calculates the derivative. apply_gradients receives pairs of gradients and their corresponding variables, then updates those variables.

Graph View