Overview
- contrastive learning learns by comparing: pull matching pairs together, push the rest apart
- core pieces: an anchor, its positive and many negatives, scored with the InfoNCE loss
- InfoNCE is a softmax that must pick the true partner, and it is tuned by the temperature and the number of negatives
- then SimCLR for images, and the limits that lead to non-contrastive methods
Contrastive learning learns representations by comparison, not by labels. Matched pairs are pulled together in the embedding space, and unmatched pairs are pushed apart. After training, the distance in the space reflects semantic similarity.
Intuition

- the anchor is one item, for example a photo of a dog
- a positive pair is two views of the same thing, for example two crops of one photo, or a photo and its caption "a dog on grass"
- the negatives are everything else in the batch, for example "a red car", a photo of a cat and "a pizza"
- training is like a matching game: which caption belongs to this photo?
InfoNCE loss
The loss for one item is:
with the similarity measured as cosine similarity:
where:
- is the embedding of the anchor item
- is the embedding of its positive partner
- is the embedding of candidate , and there are candidates in total (1 positive plus negatives)
- is the temperature, a small positive number
- is the dot product of two vectors, and is the length of vector , so the cosine similarity is between -1 and 1
- the numerator is the similarity to the one positive
- the denominator is the similarity to all candidates
How to read it:
- it is just a softmax classifier in which the "correct class" is the positive
- temperature: a small gives a sharper softmax, so the model focuses more on hard negatives (negatives that look similar to the positive)
- more negatives make the task harder, which usually gives better features
Worked example with : the similarity scores divided by are 4 for the positive and 1 and 0 for the negatives.
This loss comes from Contrastive Predictive Coding (CPC). SimCLR calls the same loss NT-Xent.
SimCLR: contrastive learning for images

- random augmentations create the positive pairs: crop and resize, flip, colour distortion, rotation, cutout, Gaussian noise, Gaussian blur and Sobel filtering
- two random augmentations of one image form a positive pair, and the other images in the batch of images are the negatives
- an encoder (a ResNet-50) gives the representation , a small MLP projection head gives , and the loss is applied on
- a non-linear projection head works clearly better than a linear head or no head
Limitations
- needs many negatives: large batches or memory banks are expensive
- false negatives: in-batch "negatives" may actually be similar, for example two different dog photos
- modality gap: image and text embeddings can occupy separate regions of the space
- coarse features: global matching of whole images can miss fine detail and object location
These limits motivate non-contrastive methods: masked modelling and self-distillation.
Note
The best-known contrastive model for images and text is CLIP.