Early Stopping

Early stopping ends training when a monitored validation score stops improving for a set number of epochs. It limits unnecessary training and can reduce overfitting.

It is one part of Hyperparameter Tuning because the stopping settings affect the final model.

Training loss and validation loss

Training loss measures error on the samples used to update the weights. Validation loss measures error on separate samples.

As training continues, training loss can fall while validation loss rises. The model is fitting its training examples more closely, while its predictions on separate examples become worse.

Monitor validation loss when the aim is to select weights that work well on new samples. Keep a separate test set for the final evaluation.

The stopping rule

The main settings are:

Setting	Meaning
Monitored value	The score checked after each epoch
Mode	min for a value that should fall; max for a value that should rise
min_delta	Required improvement over the best accepted value
patience	Allowed consecutive epochs without that improvement
Best checkpoint	A saved copy of the weights from the best accepted epoch

For loss, accept an improvement when the current loss is below best_loss - min_delta. Reset the waiting counter after an accepted improvement.

With patience=2 and min_delta=0.01:

Epoch	Validation loss	Best accepted loss	Waiting counter	Action
1	0.600	0.600	0	Save weights
2	0.500	0.500	0	Save weights
3	0.510	0.500	1	Continue
4	0.495	0.500	2	Stop and restore epoch 2

Epoch 4 improves on epoch 3. Its improvement over the best accepted loss is only 0.005, so it does not reset the counter.

Compare with the best accepted value

A comparison with only the previous epoch can reset the counter during small fluctuations and keep training for too long.

Restore the saved weights

The stopping epoch can occur after the best epoch. Restore the best checkpoint before final evaluation.

Save an independent copy of the model state. A reference to weights that keep changing cannot preserve the earlier checkpoint. Restoring weights for evaluation needs the model state. Resuming training from an exact checkpoint can also require the optimiser state.

---

Code

Keras callback

This example uses one numeric feature: positive word count minus negative word count. Each row represents one document. Positive documents have label 1; negative documents have label 0.

import numpy as np
import tensorflow as tf
 
X_train = np.array([[-3.0], [-1.0], [1.0], [3.0]], dtype="float32")
y_train = np.array([[0.0], [0.0], [1.0], [1.0]], dtype="float32")
X_val = np.array([[-2.0], [2.0]], dtype="float32")
y_val = np.array([[0.0], [1.0]], dtype="float32")
 
model = tf.keras.Sequential([
    tf.keras.Input(shape=(1,)),
    tf.keras.layers.Dense(1),
])
model.compile(
    optimizer=tf.keras.optimizers.SGD(learning_rate=0.2),
    loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
)
stopper = tf.keras.callbacks.EarlyStopping(
    monitor="val_loss",
    mode="min",
    min_delta=0.01,
    patience=3,
    restore_best_weights=True,
)
history = model.fit(
    X_train, y_train,
    validation_data=(X_val, y_val),
    epochs=200,
    batch_size=4,
    callbacks=[stopper],
    verbose=0,
)
print(len(history.history["loss"]))

Read the callback setup as follows:

• validation_data supplies the separate samples used to calculate val_loss
• Dense(1) returns one raw score per document, so the loss uses from_logits=True
• epochs=200 sets the maximum training duration
• callbacks=[stopper] attaches the stopping rule to this training run
• restore_best_weights=True restores the selected weights when training ends
• history.history["loss"] contains one training loss per completed epoch

The learning rate belongs in the optimiser. The callback monitors val_loss; loss would monitor training loss. See the Keras EarlyStopping API.

PyTorch training loop

This complete example uses the same data. It performs one full-batch update per epoch, checks validation loss, and saves each accepted checkpoint.

from copy import deepcopy
import torch
from torch import nn
 
torch.manual_seed(0)
X_train = torch.tensor([[-3.0], [-1.0], [1.0], [3.0]])
y_train = torch.tensor([[0.0], [0.0], [1.0], [1.0]])
X_val = torch.tensor([[-2.0], [2.0]])
y_val = torch.tensor([[0.0], [1.0]])
 
model = nn.Linear(1, 1)
criterion = nn.BCEWithLogitsLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.2)
 
best_loss = float("inf")
best_state = deepcopy(model.state_dict())
patience, min_delta = 3, 0.01
waiting = 0
 
for epoch in range(200):
    model.train()
    optimizer.zero_grad()
    train_loss = criterion(model(X_train), y_train)
    train_loss.backward()
    optimizer.step()
 
    model.eval()
    with torch.no_grad():
        val_loss = criterion(model(X_val), y_val).item()
 
    if val_loss < best_loss - min_delta:
        best_loss = val_loss
        best_state = deepcopy(model.state_dict())
        waiting = 0
    else:
        waiting += 1
        if waiting >= patience:
            break
 
model.load_state_dict(best_state)
model.eval()
with torch.no_grad():
    predictions = (model(X_val) >= 0).long()
print(epoch + 1, predictions.shape)

Follow the state changes:

• best_loss starts at infinity, so the first finite validation loss becomes the first checkpoint
• model.train() sets training behaviour for layers such as dropout
• zero_grad(), backward(), and step() clear old gradients, calculate new gradients, and update the weights
• model.eval() selects evaluation behaviour; torch.no_grad() separately disables gradient recording
• .item() converts the scalar loss tensor into a Python number
• deepcopy(model.state_dict()) saves an independent copy of the weights
• waiting counts consecutive epochs without sufficient improvement
• load_state_dict(best_state) restores the saved weights after the loop, including when the epoch limit is reached

The model output and labels both have shape (4, 1) during training and (2, 1) during validation. Predictions also have shape (2, 1). The threshold is zero because the model returns logits.

The independent state copy follows the PyTorch checkpoint guidance.