Overview
- a student network learns to copy the features of a teacher, which is a slow-moving (EMA) copy of the student
- no negative pairs and no pixel decoder are needed
- three examples: BYOL, data2vec and JEPA, plus why the model does not collapse
- it ends with a table that compares contrastive, masked and latent-prediction methods
Self-distillation, also called latent prediction, trains a student network to predict the features produced by a teacher network. The teacher is a slowly updated copy of the student. It needs no negative pairs and no pixel decoder.
BYOL

- the online (student) network and the target (teacher) network see two different views (augmentations) of the same image
- each network has an encoder (representation ) and a projection head (projection )
- the student has one extra predictor , which regresses the teacher's embedding
- the teacher's weights are an exponential moving average (EMA) of the student's weights
- no negative pairs are needed
where:
- is the set of student (online) weights and is the set of teacher (target) weights
- is the student's projection of view 1, and is the teacher's projection of view 2
- is the student predictor's guess of the teacher's projection
- is stop-gradient: no gradient flows into the teacher
- is the momentum, about 0.99, so the teacher changes slowly towards the student
Why it does not collapse
Collapse means the network outputs the same vector for every input. The loss would then be zero, but the features would be useless. Three parts prevent this: the predictor, the slow EMA teacher and the stop-gradient. Together they stop the two networks from agreeing in a trivial way.
data2vec: one recipe for all modalities

- the teacher sees the full input, and the student sees a masked version
- the student predicts the teacher's averaged top-layer features for the masked parts
- the loss is a regression loss (L1 or L2), and the teacher is the EMA of the student
- the same objective works for images, speech and text, with a modality-specific encoder and masking for each
JEPA: predicting in feature space

- a context encoder reads the visible part, and a predictor predicts the features of the masked part
- a target encoder (EMA) encodes the masked part, and an L2 loss compares the two
- there is no pixel reconstruction and no negatives, only feature prediction
- versions: I-JEPA (2023) for images, V-JEPA (2024) for videos, V-JEPA 2 (2025) trained on more than 1M hours of video
The full idea is in Joint-Embedding Predictive Architecture (JEPA).
Three self-supervised families compared
| Contrastive | Masked reconstruction | Latent prediction | |
|---|---|---|---|
| Predicts | which item matches | missing pixels or tokens | teacher's features of missing parts |
| Negatives | yes | no | no |
| Decoder | no | yes | no (small predictor) |
| Loss | InfoNCE | MSE or cross-entropy | L1 or L2 regression |
| Representation | coordinated | joint (for example MultiMAE) | joint or coordinated |
| Strengths | zero-shot retrieval | fine detail, dense tasks | semantic, efficient |
| Examples | SimCLR, CLIP, SigLIP | BERT, MAE, MultiMAE | BYOL, data2vec, JEPA |
Latent prediction avoids both hard-negative mining and pixel-space decoding.