Overview
- a VLA is one network that maps a camera image and an instruction straight to robot actions
- it replaces the perceive, plan and control stack, and runs closed-loop, one action per step
- main appeal: it inherits knowledge from a pretrained vision-language model, so it can handle unseen objects
- key models: RT-1, RT-2, OpenVLA, SmolVLA, pi0 and pi0.5, and V-JEPA 2, then training by behavioural cloning
A Vision-Language-Action (VLA) model is a single neural network that maps a camera observation and a natural language instruction directly to robot actions. It learns the whole pipeline, from pixels and words to motor commands, from demonstration data, instead of relying on separate hand-engineered perception, planning, and control modules.
Core Idea
A VLA replaces the modular "perceive, plan, control" stack with one learned function. At each control step it takes a fresh observation and predicts the next action, which makes it inherently closed-loop.
Inputs and output:
- camera observation of the workspace, together with the robot's proprioceptive state
- a natural language instruction such as "put the red block in the bin"
- one action per control step, usually an end-effector pose increment plus a gripper command

The VLA goes from seeing to doing. Natural language instructions give language perception, the observed scene gives visual perception, the VLA does the understanding, reasoning and decision making, and the output is an action that is executed to complete the task. The result of the action feeds back as a new observation, which allows continuous improvement. For example, the user says "Flip the pot upright" and the robot arm does it. Accurate and efficient vision-language perception is necessary for real-world embodied intelligence.
Why VLAs Are Interesting
The main appeal is generalisation to unseen objects. Because a VLA is initialised from a pretrained vision-language backbone, it inherits broad semantic knowledge and is expected to handle objects, backgrounds, and instructions it never saw during robot-specific training.

This changes how robot learning works:
- now: each robot and each task gets its own model, such as a pancake model, a box-picking model or a lipstick robot model
- in the future: one giant unsupervised or self-supervised pretrained robot foundation model learns from all the data, and each new task only needs prompting or fine-tuning, for example a trash-sorting model
Key Models
- RT-1 framed manipulation as sequence prediction over discretised actions, using a transformer trained on a large corpus of robot demonstrations
- RT-2 initialised from a pretrained vision-language model and reported emergent generalisation to novel objects and instructions
- OpenVLA is an open-weight 7-billion-parameter model trained on the Open X-Embodiment dataset, a widely used baseline
- SmolVLA is a compact 450-million-parameter model that runs on modest hardware, with asynchronous inference that decouples action prediction from execution
- pi0 and pi0.5 (2024 to 2025) use a VLM backbone plus a flow-matching action expert for smooth, high-frequency control, and pi0.5 generalises to unseen homes
- V-JEPA 2 (2025) takes a different route: it learns a video world model, then plans actions in feature space
A VLA follows the same recipe as a GUI agent built on a multimodal LLM. A camera image and an instruction go into a vision-language model, and an action head outputs tokens or a flow that become robot commands (joint angles, gripper). Only the action space changes, from mouse clicks to motor commands.
How It Is Trained
VLAs are trained by behavioural cloning on demonstrations, so data collection is the central practical bottleneck. A pretrained VLA is fine-tuned on demonstrations collected on the target robot so that its action space matches that robot and gripper.
What the data looks like:
- synchronised sequences of camera images, robot state, and actions, recorded at a fixed rate
- collected by teleoperation or by scripted expert trajectories
- fine-tuned with parameter-efficient methods when hardware is limited
VLA vs Modular Pipeline
The contrast that the comparison measures:
- the modular pipeline keeps perception, planning, and control as separate inspectable modules, so a failure can be tagged as detection-stage or grasp-stage
- the VLA entangles perception and control in one model, so it has no hard failure boundary, but its behaviour on novel objects is less predictable
- both are run closed-loop, so the comparison is like for like rather than a one-shot script against a continuously correcting policy
At deployment the VLA runs on the UR5E manipulator, where a thin mapping layer reads the current pose, applies the predicted increment, and issues the resulting target to the robot controller.