Overview

  • CLIP trains an image encoder and a text encoder so that matching images and captions land close together
  • training is one big similarity grid: the diagonal holds the matches, and each row and each column is an InfoNCE problem
  • payoff: zero-shot classification, by comparing an image with sentences such as "A photo of a {label}"
  • it ends with the models that came after: ALIGN, LAION, SigLIP and SigLIP 2

CLIP (Contrastive Language-Image Pre-training) aligns images and text in one shared space. It uses two encoders and trains them with contrastive learning on image-text pairs from the web. CLIP-ViT is the most popular vision encoder for MLLMs.

How it is trained

  • trained on 400M image-text pairs from the web, which is weak supervision
  • a text encoder maps captions and an image encoder maps photos into the same space
  • for a batch of pairs, all dot products form an N x N similarity matrix
  • the diagonal cells are the matched pairs (positives), and every other cell is a negative
  • each row and each column is one InfoNCE problem
  • the temperature is learned during training

The total loss is the average of the two directions:

where:

  • is the loss for finding the correct caption for each image (the rows)
  • is the loss for finding the correct image for each caption (the columns)

CLIP in six lines of code

# img, txt: a batch of N matched (image, caption) pairs
I = F.normalize(image_encoder(img), dim=-1)   # N x d
T = F.normalize(text_encoder(txt), dim=-1)    # N x d
logits = I @ T.t() / tau                      # N x N
labels = torch.arange(N)                      # image i <-> text i
loss = (F.cross_entropy(logits, labels) +
        F.cross_entropy(logits.t(), labels)) / 2
  • row answers: which caption matches image
  • column answers: which image matches caption
  • the batch size is the number of "classes" in each step
  • CLIP used very large batches of 32,768

So contrastive alignment is just ordinary cross-entropy on a similarity matrix.

Zero-shot classification

  1. turn each class name into a sentence: "A photo of a {label}."
  2. encode all the sentences once with the text encoder
  3. encode the test image with the image encoder
  4. predict the class whose sentence has the highest similarity to the image

There is no task-specific training, which is why it is called zero-shot. The prompt wording matters:

  • using the template instead of the bare label adds +1.3 points on ImageNet
  • ensembling 80 prompts adds about 3.5 points more

Scaling and improving contrastive models

ModelYearKey idea
CLIP2021400M image-text pairs, softmax InfoNCE over the batch
ALIGN20211.8B noisy pairs: scale can beat careful cleaning
LAION-400M and 5B2021 to 2022Open datasets filtered with CLIP (5.85B pairs in LAION-5B)
SigLIP2023Sigmoid loss: each pair is an independent match or no-match decision
SigLIP 22025Adds captioning, self-distillation and masked prediction; 109 languages
  • SigLIP does not normalise over the whole batch, because it scores each pair on its own with a sigmoid, so it is cheaper to scale to large batches
  • SigLIP 2 improves localisation and dense features, and it is the vision encoder of Qwen3-VL

Note

CLIP has a linear probe accuracy that is higher than its zero-shot accuracy, but zero-shot is the headline result because it needs no labels at all. The protocols for judging a representation are in Representation Evaluation Protocols.