This is a 7B-parameter open-source VLA, trained on 970k real-world robot demonstrations from Open-X Embodiment. It combines internet-scale vision-language pretraining with robot demonstration data, so the model can be fine-tuned instead of training a robot policy from scratch.
The Overall Architecture
OpenVLA has 3 main parts:
Raw camera image
↓
Image preprocessing
↓
Vision encoder: DINOv2 + SigLIP
↓
Vision feature projection
↓
Text instruction tokenization
↓
Vision tokens + text tokens enter Llama 2
↓
Autoregressive action-token prediction
↓
Action token IDs
↓
Action detokenization
↓
Normalized continuous action [-1, 1]
↓
Action unnormalization using dataset statistics
↓
Physical robot action
↓
Robot executes one step
OpenVLA uses a Llama 2 language model as its backbone and combines it with a visual encoder that fuses DINOv2 and SigLIP features.
| Component | Role |
|---|---|
| DINOv2 + SigLIP | Looks at the image and extracts visual information |
| Projector | Converts visual features into a form the language model can use |
| Llama 2 7B | Reasons over image + instruction and predicts action tokens |
| Action tokenizer | Converts between continuous robot actions and discrete tokens |
| Unnormalization | Converts model output back into real robot-scale movement |
The important point: OpenVLA does not output a full Python plan. It outputs low-level robot action.
RT-2 vs OpenVLA
RT-2 is mostly closed. OpenVLA goal is to create a stronger VLA open-source, with model checkpoints, fine-tuning code, and training pipeline released. OpenVLA also outperformed (according to the paper provided) across evaluated manipulation tasks while using fewer parameters.
Stage 1: Raw Input
2 Things OpenVLA receives:
- image of the current camera observations
- instructions from natural language task
Example:
- "A tabletop scene with a red cube, blue bowl, and gripper"
- "put the red cube into the blue bowl"
Stage 2: Image Processing
Raw image has to be converted to a tensor.
- image gets introduced to script via PIL image / RGB image (pillow)
- image becomes pixel_values with [batch, channels, height, width]
This includes things like:
- resizing images
- center crop / normalize
- convert to tensor
- match the vision encoder's expected input format
Stage 3: Vision Encoder
OpenVLA uses a visual encoder that fuses pretrained features from DINOv2 and SigLIP, together with a Llama 2 language model backbone.
Why we use DINOv2 and sigLIP:
- DINOv2 is good at pixel-level visual tasks such as depth estimation, semantic segmentation, that enables openVLA to understand visual structure, object shape and spatial features.
- SigLIP is good at connecting visual features with language-like concepts.
How human language is processed:
- input: "pick up the red cube"
- the vision encoder needs to produce features that represent:
- there is an object
- it is red
- it is cube-like
- it is located around this part of the image
- the gripper is here
- the bowl is there
- the vision encoder does this as vectors not as readable labels
Stage 4: Visual Features become "visual tokens"
A language model cannot naturally understand a raw image tensor. A language model understands sequences of vectors/tokens. So after the image is processed by the visual encoder, OpenVLA converts the visual features into something compatible with the Llama 2 language model.
Conceptually:
image
↓
vision encoder
↓
visual feature vectors
↓
projector (role: translates visual features into LLM's internal vector format)
↓
LLM-compatible visual tokens
Stage 5: Text Instruction Tokenization
The language instructions also needs to be tokenized. The user instruction containing human words ("put the red cube into the blue bowl") gets internally formatted by OpenVLA ("what actions should the robot take to put the red cube into the blue bowl?").
The OpenVLA code builds this prompt inside predict_action, lowercases the instruction, tokenizes it and prepares it for the model.
The text becomes token IDs:
"What" → token ID
"action" → token ID
"should" → token ID
...
"bowl" → token ID
Now, the model has 2 streams:
- visual tokens from image
- text tokens from instructions
Stage 6: Vision-language Fusion inside the LLM
The LLM receives a combined context of [visual tokens] + [text instruction tokens], where it attends to both.
This means that the LLM can connect:
- 'red' --> visual area containing red object
- 'cube' --> visual shape of cube
- 'bowl' --> visual object that looks like a bowl
It is not an object detector that creates bounding boxes. Instead, it forms an internal representation like:
- the instruction refers to this object in the image
- the robot should close when near it
- the gripper should close when near it
- the move towards the target container
Stage 7: LLM's Output
A normal LLM outputs text tokens:
| Input | "The capital of France is" |
|---|---|
| Output | "Paris" |
However, OpenVLA's output action tokens:
| Input | image + "pick up the red cube" |
|---|---|
| Output | [action_token_1, action_token_2, action_token_3, ..., action_token_7] |
For a 7 degree of freedom (7-DoF) action, those 7 tokens correspond to:
- dx
- dy
- dz
- rotation x (roll-like movement)
- rotation y (pitch-like movement)
- rotation z (yaw-like movement)
- gripper command
Therefore, instead of predicting:
"move left and close gripper"
It predicts something like:
[173, 128, 140, 130, 122, 128, 250]
Stage 8: Why actions become tokens
Robots use continuous numbers. For example, we use dy = 0.012m, etc. But LLMs naturally output discrete tokens. So OpenVLA converts continuous actions into discrete bins.
The OpenVLA ActionTokenizer uses 256 bins by default, clips action values between -1 and 1, discretizes them, and maps them into the least-used tokens at the end of the tokenizer vocabulary.
Conceptually:
continuous action value:
dx = 0.23
normalize/clamp:
dx is inside [-1, 1]
discretize:
dx falls into bin 157
map to token:
bin 157 → action token ID
Normalized action:
[ 0.20, -0.10, 0.35, 0.00, 0.05, -0.02, 1.00 ]
Tokenized action:
[ token_154, token_116, token_172, token_128, token_134, token_125, token_255 ]
Stage 9: Action Embedding
There are 2 related ideas:
- Action token ID (an integer from 0-255 that can be converted to a vector embedding)
- Action token embedding (basically a vector)
Action Token ID is the discrete symbol selected by the model (e.g. token_154). However, if we are currently in the intermediate step, and we have an intermediate generated token used to predict the next action token, then we would need to convert this action token back to action vector embedding to be fed back into the transformer to get the next action token.
Once all action token IDs are created (the full 7), then we are done and this will be passed to a 'lookup table' that converts these action token IDs to continuous movement value that can be directly passed to the TCP.
Stage 10: Autoregressive Action Prediction
The above feature for predicting 1 token at a time, and then converting them to vector embedding, passing it back to the model to predict the next action token, is the Autoregressive Action Prediction nature of the models.
Autoregressive means:
predict token 1 then predict token 2 conditioned on token 1then predict token 3 conditioned on token 1 and token 2...
Note: It is not predicting a whole long trajectory. Instead, it is predicting one action vector for the current observation.
Stage 11: Action Detokenization
From this:
predicted_action_token_ids = [token_154, token_116, token_172, token_128, token_134, token_125, token_255]
OpenVLA converts them back into normalized continuous values using:
normalized_actions = action_tokenizer.decode_token_ids_to_actions(...)The action tokenizer converts token IDs back into bin centers, producing values in the normalized action range.
Conceptually:
token_154 → 0.20token_116 → -0.10token_172 → 0.35...
So now we have:
normalized action:[0.20, -0.10, 0.35, 0.00, 0.05, -0.02, 1.00]
But these values are still normalized. They are not yet real robot-scale commands.
Stage 12: Action Unnormalization
During training, robot actions from different datasets are normalized into a common range, usually around:
[-1, 1]
But at execution time, the robot needs physical values.
For example:
normalized dx = 0.20
does not directly mean:
move 0.20 metres
It means:
0.20 relative to the action distribution of the dataset
OpenVLA uses dataset-specific normalization statistics. In the code, it retrieves q01 and q99, which are approximately the 1st and 99th percentile action values for that dataset, then maps normalized actions back into the real action scale.
The code formula is essentially:
physical_action =0.5 × (normalized_action + 1) × (q99 - q01) + q01So:
normalized -1 → near q01normalized 0 → middle of q01 and q99normalized +1 → near q99
This is why unnorm_key matters.
From the repository:
action = vla.predict_action(
**inputs,
unnorm_key="bridge_orig",
do_sample=False
)The unnorm_key tells the model which dataset’s action statistics to use. For UR5E, we need UR5E-specific action normalization and unnormalization.
Stage 13: Final Action sent to robot
After unnormalization, OpenVLA returns something like:
action = [dx, dy, dz, rx, ry, rz, gripper]
For UR5e, a possible mapping would be:
dx, dy, dz: small TCP translation commanddRx, dRy, dRz: small TCP orientation changegripper: open / close command
But UR5e uses pose format, so must choose the right action representation:
- Cartesian delta pose
- Absolute TCP pose
- Joint delta
- End-effector velocity
- Normalized action in some dataset convention
Stage 14: Runtime control loop
One OpenVLA prediction gives one action.
So actual robot deployment looks like a loop:
while task not done:
1. Capture latest image
2. Keep same instruction: "put the red cube into the blue bowl"
3. Run OpenVLA
4. Get action: [dx, dy, dz, dRx, dRy, dRz, gripper]
5. Convert action into UR5e command
6. Execute small movement
7. Observe again
So it is still closed-loop in the sense that the model repeatedly sees new images after movement.
Comparing both modular and VLA pipeline:
MODULAR: re-detect object → recompute 3D position → update pick/place primitive
OpenVLA: re-observe image → predict next learned action
Stage 15: Training Pipeline
A training sample looks like:
- observation image at time t
- language instruction
- robot action taken at time t
Example:
Image: gripper above table, red cube visible
Instruction: "pick up the red cube"
Action: move slightly forward, slightly down, close gripper later
The training process is:
raw action
↓
normalize action using dataset statistics
↓
discretize action into bins
↓
convert bins into action tokens
↓
feed image + instruction into VLA
↓
train model to predict the correct action tokens
So the model is trained by imitation.
It is learning:
When the scene looks like this, and the instruction says this,the demonstrator moved like this.
It is not learning from reward. It is not planning with a physics engine. It is supervised imitation learning over many robot demonstrations. OpenVLA was trained on a large collection of real robot demonstrations from Open X-Embodiment, and the released main model was trained on a mixture spanning 970k trajectories.
Stage 16: Training-time action token example
Suppose your raw robot action is:
actual action:
dx = 0.006
mdy = -0.002
mdz = 0.010
mdRx = 0.001
dRy = 0.000
dRz = -0.002
gripper = close
First, normalize it:
normalized action:
[0.18, -0.05, 0.30, 0.02, 0.00, -0.04, 1.00]
Then discretize:
bin indices:
[151, 121, 166, 130, 128, 123, 255]
Then map to action tokens:
action tokens:
[token_151, token_121, token_166, token_130, token_128, token_123, token_255]
The training target becomes:
Given: image + "pick up the red cube"
Predict: token_151 token_121 token_166 token_130 token_128 token_123 token_255
This is why OpenVLA can use a language-model-style objective.
It is basically next-token prediction, except the “next tokens” represent robot motion.
Stage 17: Weakness of OpenVLA
Main weakness is the continuous to discretized actions.
Another weakness is not using Action Chunking.
Base OpenVLA predicts one action per forward pass. It is fully autoregressive, closed-loop every step. It does not chunk.
SmolVLA belongs to a newer generation of VLAs that predicts a short sequence of future action tokens per inference (action chunking), not one action.
Chunking improves smoothness and inference efficiency (fewer LLM calls per unit of motion) but reduces reactivity. The robot commits to executing several steps open-loop before re-observing.
longer chunks = smoother/more efficient but slower to react to perturbation
shorter chunks = more reactive but noisier
This is directly comparable to Architecture A's closed-loop re-perception design, which re-observes after every single action. Chunk size is effectively a tunable reactivity/efficiency knob unique to Architecture B.
Stage 18: Mapping to UR5e
For a UR5e VLA dataset, each training step might look like:
sample = {
"image": camera_frame_t,
"instruction": "pick up the red cube",
"robot_state": {
"joint_positions": q_t,
"tcp_pose": pose_t,
"gripper": gripper_t,
},
"action": {
"delta_tcp_pose": pose_t_plus_1 - pose_t,
"gripper_action": gripper_t_plus_1,
}
}For OpenVLA-style learning, the common idea is end-effector delta action:
[Δx, Δy, Δz, Δrotation, gripper]
Stage 19: Full Pipeline Summary
┌───────────────────────────┐
│ Camera image │
│ RGB scene observation │
└─────────────┬─────────────┘
↓
┌───────────────────────────┐
│ Image preprocessing │
│ resize, normalize, tensor │
└─────────────┬─────────────┘
↓
┌───────────────────────────┐
│ Vision encoder │
│ DINOv2 + SigLIP │
└─────────────┬─────────────┘
↓
┌───────────────────────────┐
│ Projector │
│ visual features → LLM dim │
└─────────────┬─────────────┘
│
┌──────────────────────┐ │
│ Language instruction │ │
│ "pick red cube" │ │
└───────────┬──────────┘ │
↓ │
┌──────────────────────┐ │
│ Text tokenizer │ │
│ words → token IDs │ │
└───────────┬──────────┘ │
↓ ↓
┌─────────────────────────────────┐
│ Llama 2 VLM backbone │
│ attends to image + instruction │
└────────────────┬────────────────┘
↓
┌─────────────────────────────────┐
│ Autoregressive generation │
│ predicts action tokens │
└────────────────┬────────────────┘
↓
┌─────────────────────────────────┐
│ Action tokenizer decode │
│ token IDs → normalized action │
└────────────────┬────────────────┘
↓
┌─────────────────────────────────┐
│ Unnormalization │
│ normalized → robot-scale action │
└────────────────┬────────────────┘
↓
┌─────────────────────────────────┐
│ Robot controller │
│ execute one motion step │
└─────────────────────────────────┘