Distributed Data Parallel trains one model on several GPUs at once. Each GPU holds a full copy of the model, processes its own share of each batch, and the gradients are averaged across GPUs so every copy stays identical. It changes the effective Batch Size and Epochs.
How One Step Works
- one process per GPU, each with the same initial weights
- a DistributedSampler gives each process a different part of the dataset
- each process runs forward and backward on its own mini-batch
- during backward, gradients are averaged across processes with an all-reduce (often NCCL on NVIDIA GPUs)
- every process applies the same averaged gradient, so the weights stay the same everywhere
Effective Batch Size
- a larger effective batch may need a higher learning rate or a warm-up
SyncBatchNorm
- batch norm computes mean and variance over the batch on each GPU separately
- with small per-GPU batches (for example 6 large images), those statistics are noisy
- SyncBatchNorm computes them across all GPUs, which makes them stable
- the cost is extra communication in every batch norm layer
DDP Versus DataParallel
- the older
DataParalleluses one process and copies the model to each GPU every step, which is slower - DDP uses one process per GPU and overlaps gradient communication with the backward pass
Practical Points
- call
sampler.set_epoch(epoch)so each epoch shuffles differently - save checkpoints and write logs from rank 0 only
- metrics must be gathered from all processes before they are reported