Weight Decay

Weight decay reduces weight values during training. It is a regularisation method that can reduce overfitting. Its strength is a hyperparameter.

Dropout changes which activations are used during training. Weight decay acts on the model's parameters.

L2 penalty and decoupled weight decay

Method	Where it acts	What it does
L2 regularisation	Loss	Adds a penalty based on squared weights
Decoupled weight decay	Optimiser update	Shrinks weights separately from the loss gradient

One common L2 convention is:

$$L_{total}=L_{data}+\lambda \sum _i{w}_i^2$$

The symbols mean:

• $L_{data}$ measures prediction error
• $w_i$ are the selected weights
• $\lambda$ controls penalty strength

For weights $[2,-1]$ and $\lambda =0.01$, the penalty is $0.01(4+1)=0.05$. If the data loss is $0.30$, the total is $0.35$.

With plain SGD, an L2 penalty can produce the same update as weight decay after matching the coefficients. Adaptive optimisers change this relationship. AdamW applies weight decay separately from Adam's gradient history. See the AdamW paper.

A stronger penalty pushes more strongly toward small weights. An excessive value can cause underfitting. Compare validation performance when selecting it.

Code

PyTorch: AdamW

import torch
 
w = torch.nn.Parameter(torch.tensor([2., -1.]))
optimizer = torch.optim.AdamW([w], lr=0.1, weight_decay=0.01)
 
optimizer.zero_grad()
loss = (w * 0).sum()
loss.backward()
optimizer.step()
print(w.detach().tolist())

The result is approximately [1.998, -0.999].

Read the update:

• [w] selects the parameters to decay
• loss is deliberately zero, with zero gradients, so the example isolates weight decay
• weight_decay=0.01 sets the decay coefficient
• The decay multiplies each weight by 1 - 0.1 * 0.01, which is 0.999
• detach() removes the gradient tracking connection before reading the result

For a full model, use torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01). The selected parameters then also receive updates from the prediction loss.

To add the L2 penalty directly instead:

penalty = 0.01 * w.square().sum()
print(round(penalty.item(), 3))

This gives about 0.05. Add penalty to the data loss before backward() when implementing L2 regularisation. Keep optimiser weight decay at zero when the intended method is the explicit L2 penalty alone.

TensorFlow and Keras: layer L2 penalty

import tensorflow as tf
 
layer = tf.keras.layers.Dense(
    1,
    use_bias=False,
    kernel_initializer=tf.keras.initializers.Constant([[2.], [-1.]]),
    kernel_regularizer=tf.keras.regularizers.L2(0.01),
)
output = layer(tf.constant([[1., 1.]]))
print(round(float(layer.losses[0].numpy()), 3))

0.05

Read the layer:

• The input has 2 features, so the kernel contains 2 weights
• kernel_regularizer applies the penalty to the kernel weights
• Calling layer(...) builds the layer and produces its output
• layer.losses[0] contains the regularisation penalty
• A model's standard fit() loop includes these layer penalties in its training loss

The Keras L2 API uses the coefficient convention in the equation above.

For decoupled weight decay, configure an optimiser instead:

optimizer = tf.keras.optimizers.AdamW(
    learning_rate=1e-3, weight_decay=0.01
)

Pass this optimiser to model.compile(). Its weight_decay setting acts during the update. A layer's kernel_regularizer adds a loss penalty. These are separate controls.