A world model predicts how the world will change. In robotics, it learns the consequence of a proposed action rather than directly deciding which action the robot must execute.
Policy compared with world model
| Model | Main mapping | Deployment behaviour |
|---|---|---|
| VLA policy | observation + instruction → action | Sends an action or action chunk to the robot |
| World model | current latent state + candidate action → future latent state | Lets a planner compare possible consequences before execution |
RT-1, RT-2, OpenVLA, and SmolVLA are policies. They learn to output actions from observations and instructions. A world model instead learns cause and effect:
current state + candidate action → predicted next statePlanning loop
- Propose several candidate action sequences
- Use the world model to perform a Latent Rollout for each sequence
- Score the predicted result against the goal
- Select the best sequence
- Execute the first action
- Observe the new state and plan again
The Cross-Entropy Method is one way to search the candidate sequences.
Generative and predictive world models
- A generative world model predicts pixels or visual tokens that can be decoded into pixels
- A predictive world model predicts latent representations and avoids pixel generation
V-JEPA 2 and GeoWorld are predictive world models. Their planner compares predicted future representations with a goal representation.
Important distinction
A world model is a role, while Joint-Embedding Predictive Architecture (JEPA) is an architecture family. A JEPA can support a world model, but the two terms do not mean the same thing.
My way of understanding it
The clearest comparison for me is with SmolVLA. SmolVLA is a policy that directly maps its observations and instructions to an action chunk, which is then output to the physical robot. However, V-JEPA 2-AC is a world model that takes a candidate action sequence and predicts the future latent representation that would result from those actions.
In a way, we are trying to let the model understand the physics of our world and learn cause and effect through experience. Instead of immediately asking, "What action should I execute?", the world model asks, "If I take this action, what will happen next?"
This is why I think of a policy as the model that acts and a world model as the model that imagines the consequence before acting.