Energy-Based Model

An energy-based model gives a scalar score to a proposed relationship or outcome. A low energy means that the proposal is more compatible. A high energy means that it is a worse fit.

Energy landscape

An energy landscape is the set of energy scores across many possible candidates. In planning, each point can represent a candidate future state or action sequence.

candidate 1 → low energy  → good match with the goal
candidate 2 → high energy → poor match with the goal

This turns inference into an optimization or search problem. The planner searches for the candidate with the lowest energy.

Energy compared with loss

Energy does not replace the training loss.

• Energy is the compatibility score that the model assigns to a candidate
• Loss is the training objective that adjusts the model parameters so that good candidates receive lower energy than bad candidates

Use in world models

An energy-based World Model can score any proposed future instead of producing one fixed answer. This is useful when several future actions are plausible.

In V-JEPA 2, the goal-conditioned energy is the distance between the predicted future representation and the goal representation. In GeoWorld, this becomes a hyperbolic geodesic distance in a Hyperbolic Latent Space.

The Cross-Entropy Method searches action sequences that minimize this energy.

My way of understanding energy

We have a scalar energy function that measures how closely related, or compatible, two states are. For example, the current state and a sensible next state have low energy, which means they are more compatible. The current state and a poor next state have higher energy, which means they are less compatible.

I think of it as the more energy it requires to go from state $s_t$ to $s_{t+1}$, the worse the option is. Since one state can have multiple plausible next states, we need to find the best combination of actions, latent states, or next states that gives us the minimum energy.

My initial way of thinking about this was that energy replaces the traditional error or loss. The correction is that energy is the score assigned to a proposed outcome, while the training loss teaches the model how to assign useful energy scores.

This is why the inference pass becomes an optimization or search problem. We can score any proposed prediction and search for the path of least energy.