V-JEPA 2

V-JEPA 2 is a self-supervised video model that learns representations for understanding, prediction, and planning. It learns from masked video prediction instead of generating pixels.

Base V-JEPA 2

The base model receives partially masked video and predicts the representation of the hidden video regions. This extends the masked-image idea in Joint-Embedding Predictive Architecture (JEPA) to space and time.

The base model learns useful visual and temporal representations, but it does not directly model the causal effect of a robot action.

V-JEPA 2-AC

V-JEPA 2-AC adds a frame-causal action-conditioned predictor on top of the frozen V-JEPA 2 encoder.

current image → frozen encoder → current latent state
robot state  ───────────────────────────────┐
candidate action ───────────────────────────┤
                                             ↓
                           action-conditioned predictor
                                             ↓
                              predicted future latent state

The predictor receives the current visual representation, the end-effector state, and a candidate action. It predicts the future visual representation that would result from that action.

Training

• Teacher forcing uses the real current state to train one-step prediction
• Latent Rollout loss feeds a predicted state back into the predictor to train multi-step prediction
• The visual encoder stays frozen while the action-conditioned predictor learns from robot interaction data

Planning

The model receives a current image and a goal image. The Cross-Entropy Method proposes action sequences, and V-JEPA 2-AC predicts the final latent state for each sequence. The planner selects the sequence whose predicted state is closest to the goal representation.

Only the first action is executed. The robot observes the new state and plans again, so the process is closed-loop model-predictive control.

Compared with a VLA

SmolVLA directly maps observations and an instruction to an action chunk. V-JEPA 2-AC takes a candidate action sequence and predicts its consequence. The first is a policy, while the second is a World Model.

Limitations

• prediction errors accumulate during long autoregressive rollouts
• the action search space grows rapidly with the planning horizon
• camera position affects the meaning of end-effector actions when the camera is not calibrated
• visual goals are less natural than language goals for many tasks

GeoWorld keeps the action-conditioned predictive idea and targets the first two long-horizon problems with hyperbolic geometry and Geometric Reinforcement Learning.

My way of understanding V-JEPA 2-AC

While we have a V-JEPA 2 model that is able to make predictions on masked portions of the video, we still need to incorporate a way for the model to learn how taking certain actions will change the state of the world.

To solve this, the V-JEPA 2 weights are frozen and a frame-causal action-conditioned predictor is trained on top. It predicts future video representations from the current representation, robot state, and candidate actions.

During planning, CEM proposes several candidate action sequences. These sequences go into V-JEPA 2-AC, and we select the sequence that produces a latent representation closest to the goal.

At a high level, I first thought this sounded similar to next-token prediction with a greedy top-k of 1. The important correction is that CEM evaluates a population of full action sequences and repeatedly refines the distribution. It does not make one greedy choice and stop.

The main link to GeoWorld is the long-horizon issue. Autoregressive prediction errors accumulate, and the number of possible action sequences grows rapidly. GeoWorld builds on this action-conditioned idea and tries to make the multi-step predictions more stable.