Overview
- JEPA learns by predicting the representation of the hidden part, not its pixels
- two paths: a context path that makes the prediction, and a target path (EMA encoder) that gives the answer to compare with
- versions: I-JEPA for images, V-JEPA for video and V-JEPA 2 at large scale
- it belongs to the self-distillation family, and it suits planning because it compares high-level states
A Joint-Embedding Predictive Architecture, or JEPA, learns by predicting representations instead of reconstructing every pixel. The model receives visible context and predicts the latent representation of hidden or future content.
Core idea
My own Summary:
Unlike generative models that try to predict raw pixels or tokens, JEPA learns to predict representations in latent space.
During training, there are two parallel paths:
- Context (prediction) path
- Target path
where:
- represents an encoder
- represents the predictor
- and represent data in the observation/input space, such as an image, image patches, text tokens, or video frames
- represents an encoded latent representation
Context or the prediction path
- We give it the input image or text,
- will then pass through the context encoder, resulting in
- This passes through a predictor that predicts the target latent representation
Target path
- This path will convert the given target into via another encoder called the target encoder
- We then compare our predicted and
- Note that this comparison is in the latent space since both prediction and target are representations in latent space (that's the speciality of JEPA)
Few key points
- The encoders can be transformer based
- The predictor can also be a transformer based model
- The prediction loss is backpropagated through the predictor and context encoder, updating their weights
- The target encoder does not receive gradients from this loss
For EMA-based JEPA implementations such as I-JEPA
- The target encoder gets its weights updated via exponential moving average (EMA) that takes in a weighted combination of weights from both the current target encoder and the current context encoder
- EMA updates each target-encoder parameter using the corresponding context-encoder parameter meaning the two encoders need matching parameter structures for this to work
JEPA encourages the representation to keep useful structure and semantics. It can ignore pixel details that do not help the prediction.
Main forms
- I-JEPA predicts representations of masked image regions from visible image context
- V-JEPA extends the idea to masked regions across video and learns temporal representations
- V-JEPA 2 scales video pretraining and adds an action-conditioned world model for planning
- GeoWorld maps the visual representations into a Hyperbolic Latent Space and adds geometry-aware rollout refinement
The dates are I-JEPA (2023) for images, V-JEPA (2024) for videos and V-JEPA 2 (2025) trained on more than 1M hours of video.
Place among self-supervised methods
JEPA belongs to the latent prediction (self-distillation) family:
- the context encoder reads the visible part, and the predictor predicts the features of the masked part
- the target encoder is an EMA copy, and an L2 loss compares the predicted and target features
- there is no pixel reconstruction and no negatives, only feature prediction
- so JEPA avoids both the hard-negative mining of contrastive learning and the pixel decoding of masked autoencoders
Why it helps planning
Pixel generation spends capacity on texture, lighting, and other low-level details. A predictive JEPA can compare higher-level latent states instead. This makes it suitable for an Energy-Based Model that scores whether a predicted future is compatible with a goal.
Important distinction
JEPA is not another name for every World Model. JEPA describes how representations are learned and predicted. A world model describes the role of predicting future consequences.
My way of understanding JEPA
The whole idea of JEPA is that it is a model that does not generate pixels like a generic diffusion model. Observations are encoded into a smaller dimensional space, which is then used to make predictions that minimize the energy between the encoded goal and the predicted observation.
The benefit of making predictions in latent space instead of purely on pixels is that the model does not rely only on low-level pattern matching or pixel-level similarities. Instead, it can learn higher-level representations that contain hidden structure and temporal dependencies.
My comparison is with word embeddings. Words with similar meanings are closer to one another in an embedding space. In the same way, visually or temporally related states can be closer to one another in the JEPA representation space.
I-JEPA made the idea clearer for me. Part of the image is masked while other context-rich parts stay visible. The model predicts the masked part in latent representation instead of reconstructing every pixel. V-JEPA then applies this idea to another modality, which is video.