Overview

  • a student network learns to copy the features of a teacher, which is a slow-moving (EMA) copy of the student
  • no negative pairs and no pixel decoder are needed
  • three examples: BYOL, data2vec and JEPA, plus why the model does not collapse
  • it ends with a table that compares contrastive, masked and latent-prediction methods

Self-distillation, also called latent prediction, trains a student network to predict the features produced by a teacher network. The teacher is a slowly updated copy of the student. It needs no negative pairs and no pixel decoder.

BYOL

  • the online (student) network and the target (teacher) network see two different views (augmentations) of the same image
  • each network has an encoder (representation ) and a projection head (projection )
  • the student has one extra predictor , which regresses the teacher's embedding
  • the teacher's weights are an exponential moving average (EMA) of the student's weights
  • no negative pairs are needed

where:

  • is the set of student (online) weights and is the set of teacher (target) weights
  • is the student's projection of view 1, and is the teacher's projection of view 2
  • is the student predictor's guess of the teacher's projection
  • is stop-gradient: no gradient flows into the teacher
  • is the momentum, about 0.99, so the teacher changes slowly towards the student

Why it does not collapse

Collapse means the network outputs the same vector for every input. The loss would then be zero, but the features would be useless. Three parts prevent this: the predictor, the slow EMA teacher and the stop-gradient. Together they stop the two networks from agreeing in a trivial way.

data2vec: one recipe for all modalities

  • the teacher sees the full input, and the student sees a masked version
  • the student predicts the teacher's averaged top-layer features for the masked parts
  • the loss is a regression loss (L1 or L2), and the teacher is the EMA of the student
  • the same objective works for images, speech and text, with a modality-specific encoder and masking for each

JEPA: predicting in feature space

  • a context encoder reads the visible part, and a predictor predicts the features of the masked part
  • a target encoder (EMA) encodes the masked part, and an L2 loss compares the two
  • there is no pixel reconstruction and no negatives, only feature prediction
  • versions: I-JEPA (2023) for images, V-JEPA (2024) for videos, V-JEPA 2 (2025) trained on more than 1M hours of video

The full idea is in Joint-Embedding Predictive Architecture (JEPA).

Three self-supervised families compared

ContrastiveMasked reconstructionLatent prediction
Predictswhich item matchesmissing pixels or tokensteacher's features of missing parts
Negativesyesnono
Decodernoyesno (small predictor)
LossInfoNCEMSE or cross-entropyL1 or L2 regression
Representationcoordinatedjoint (for example MultiMAE)joint or coordinated
Strengthszero-shot retrievalfine detail, dense taskssemantic, efficient
ExamplesSimCLR, CLIP, SigLIPBERT, MAE, MultiMAEBYOL, data2vec, JEPA

Latent prediction avoids both hard-negative mining and pixel-space decoding.