Learning Rate

The learning rate controls the scale of a parameter update during training. It is a hyperparameter supplied to Optimisers.

Effect on training

Learning rate	Possible result
Too large	Updates pass over a useful solution, and loss can rise or oscillate
Too small	The model makes slow progress within the available training time
Suitable	Training makes steady progress toward lower loss

A rate such as 0.001 is a starting value to test. The suitable value depends on the model, data, and optimiser.

One update

For plain SGD:

$$w_{new}=w-\eta g$$

The symbols mean:

• $w$ is the current parameter
• $g$ is its gradient
• $\eta$ is the learning rate

Suppose $w=2$ and $g=0.4$:

Learning rate	Update size	New parameter
$0.1$	$0.1\times 0.4=0.04$	$1.96$
$0.01$	$0.01\times 0.4=0.004$	$1.996$

Adam also uses gradient history to scale its updates. Its learning rate still controls the overall update scale.

Code

PyTorch

This example performs the first row of the table:

import torch
 
w = torch.nn.Parameter(torch.tensor(2.0))
optimizer = torch.optim.SGD([w], lr=0.1)
 
optimizer.zero_grad()
loss = 0.4 * w
loss.backward()
optimizer.step()
print(round(w.item(), 3))

1.96

Read the update:

• Parameter makes w trainable
• [w] is the list of parameters passed to the optimiser
• The derivative of 0.4 * w is 0.4, so backward() stores this value in w.grad
• lr=0.1 makes step() subtract 0.04 from w

For a full model, the same setting is torch.optim.Adam(model.parameters(), lr=1e-3).

TensorFlow and Keras

import numpy as np
import tensorflow as tf
 
X = np.array([[0.], [1.]], dtype="float32")
y = np.array([[0.], [2.]], dtype="float32")
model = tf.keras.Sequential([
    tf.keras.Input(shape=(1,)),
    tf.keras.layers.Dense(1),
])
optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3)
model.compile(optimizer=optimizer, loss="mse")
model.fit(X, y, batch_size=2, epochs=1, verbose=0)

Read the settings:

• X contains 2 input samples and y contains their target values
• learning_rate=1e-3 belongs to the optimiser constructor
• compile() gives the model this configured optimiser
• fit() receives training settings such as batch_size and epochs

The current Keras optimiser API accepts learning_rate. The fit API accepts the data and training settings.

To change an existing constant learning rate before further training:

optimizer.learning_rate.assign(1e-4)

This changes the rate from 0.001 to 0.0001. The model keeps its learned weights.