Dropout and Coadaptation

Problem

Co-adaptation happens when neurons become overly reliant on each other. When one neuron fires, and another simply copies it rather than learning independently. This leads to overfitting where the network memorises the training set instead of generalising.

Analogy: Imagine a smart classmate sitting next to us. Every test, we wanna copy his answers. Because of this, we never actually learn the material.

Solution

Dropout is a regularisation technique that randomly turns off a fraction of neurons during each training step. Because any neuron might be absent, every neuron is forced to learn on its own, thereby reducing co-adaptation.

Analogy: Now our smart classmate might randomly be absent on a test day. We can no longer rely on copying him. Therefore we have to learn the material ourselves.

How It Works

During training, each neuron is independently zeroed out with probability p (commonly 0.5):

Layer output (no dropout):  [0.9,  0.4,  0.7,  0.2,  0.8]
Dropout mask (p = 0.5):     [  1,    0,    1,    0,    1 ]
Layer output (after masking):[0.9, 0.0, 0.7, 0.0, 0.8]
Scaled training output:      [1.8, 0.0, 1.4, 0.0, 1.6]

During training, standard PyTorch and Keras dropout scale retained values by 1 / (1 - p). With p=0.5, retained values double. This preserves their expected value across random masks.

During inference, the dropout layer passes its input through unchanged. PyTorch Dropout reference.

Benefits

Benefit	Explanation
Reduces overfitting	Neurons learn robust, independent features
Reduces co-adaptation	No neuron can rely on any specific other neuron
Acts as Ensemble Methods (Bagging, Boosting)	Each training step trains a slightly different sub-network

Code

PyTorch: training and evaluation modes

import torch
from torch import nn
 
x = torch.ones(8)
dropout = nn.Dropout(p=0.5)
dropout.train()
training_output = dropout(x)
dropout.eval()
evaluation_output = dropout(x)
print(training_output)    # Each value is either 0 or 2
print(evaluation_output)  # Eight values of 1

Read the behaviour:

p=0.5 is the probability of removing each value, rather than an exact fraction removed on every call
train() enables random masking and scaling
eval() makes this layer pass its input through unchanged
model.train() and model.eval() apply the mode to registered layers throughout a model
torch.no_grad() controls gradient recording separately; it does not set evaluation mode

TensorFlow/Keras

import tensorflow as tf
 
x = tf.ones((1, 8))
dropout = tf.keras.layers.Dropout(rate=0.5)
training_output = dropout(x, training=True)
evaluation_output = dropout(x, training=False)
print(training_output.numpy())    # Values are 0 or 2
print(evaluation_output.numpy())  # Values are all 1

rate has the same role as PyTorch's p. A standard fit() call sets training behaviour during updates and evaluation behaviour during validation. The output shape remains equal to the input shape.

JJ

Explorer