Overview
- CLIP trains an image encoder and a text encoder so that matching images and captions land close together
- training is one big similarity grid: the diagonal holds the matches, and each row and each column is an InfoNCE problem
- payoff: zero-shot classification, by comparing an image with sentences such as "A photo of a {label}"
- it ends with the models that came after: ALIGN, LAION, SigLIP and SigLIP 2
CLIP (Contrastive Language-Image Pre-training) aligns images and text in one shared space. It uses two encoders and trains them with contrastive learning on image-text pairs from the web. CLIP-ViT is the most popular vision encoder for MLLMs.
How it is trained

- trained on 400M image-text pairs from the web, which is weak supervision
- a text encoder maps captions and an image encoder maps photos into the same space
- for a batch of pairs, all dot products form an N x N similarity matrix
- the diagonal cells are the matched pairs (positives), and every other cell is a negative
- each row and each column is one InfoNCE problem
- the temperature is learned during training
The total loss is the average of the two directions:
where:
- is the loss for finding the correct caption for each image (the rows)
- is the loss for finding the correct image for each caption (the columns)
CLIP in six lines of code
# img, txt: a batch of N matched (image, caption) pairs
I = F.normalize(image_encoder(img), dim=-1) # N x d
T = F.normalize(text_encoder(txt), dim=-1) # N x d
logits = I @ T.t() / tau # N x N
labels = torch.arange(N) # image i <-> text i
loss = (F.cross_entropy(logits, labels) +
F.cross_entropy(logits.t(), labels)) / 2- row answers: which caption matches image
- column answers: which image matches caption
- the batch size is the number of "classes" in each step
- CLIP used very large batches of 32,768
So contrastive alignment is just ordinary cross-entropy on a similarity matrix.
Zero-shot classification

- turn each class name into a sentence: "A photo of a {label}."
- encode all the sentences once with the text encoder
- encode the test image with the image encoder
- predict the class whose sentence has the highest similarity to the image
There is no task-specific training, which is why it is called zero-shot. The prompt wording matters:
- using the template instead of the bare label adds +1.3 points on ImageNet
- ensembling 80 prompts adds about 3.5 points more
Scaling and improving contrastive models
| Model | Year | Key idea |
|---|---|---|
| CLIP | 2021 | 400M image-text pairs, softmax InfoNCE over the batch |
| ALIGN | 2021 | 1.8B noisy pairs: scale can beat careful cleaning |
| LAION-400M and 5B | 2021 to 2022 | Open datasets filtered with CLIP (5.85B pairs in LAION-5B) |
| SigLIP | 2023 | Sigmoid loss: each pair is an independent match or no-match decision |
| SigLIP 2 | 2025 | Adds captioning, self-distillation and masked prediction; 109 languages |
- SigLIP does not normalise over the whole batch, because it scores each pair on its own with a sigmoid, so it is cheaper to scale to large batches
- SigLIP 2 improves localisation and dense features, and it is the vision encoder of Qwen3-VL
Note
CLIP has a linear probe accuracy that is higher than its zero-shot accuracy, but zero-shot is the headline result because it needs no labels at all. The protocols for judging a representation are in Representation Evaluation Protocols.