A Vision-Language-Action (VLA) model is a single neural network that maps a camera observation and a natural language instruction directly to robot actions. It learns the whole pipeline, from pixels and words to motor commands, from demonstration data, instead of relying on separate hand-engineered perception, planning, and control modules.
Core Idea
A VLA replaces the modular "perceive, plan, control" stack with one learned function. At each control step it takes a fresh observation and predicts the next action, which makes it inherently closed-loop.
Inputs and output:
- camera observation of the workspace, together with the robot's proprioceptive state
- a natural language instruction such as "put the red block in the bin"
- one action per control step, usually an end-effector pose increment plus a gripper command
Why VLAs Are Interesting
The main appeal is generalisation to unseen objects. Because a VLA is initialised from a pretrained vision-language backbone, it inherits broad semantic knowledge and is expected to handle objects, backgrounds, and instructions it never saw during robot-specific training.
Key Models
- RT-1 framed manipulation as sequence prediction over discretised actions, using a transformer trained on a large corpus of robot demonstrations
- RT-2 initialised from a pretrained vision-language model and reported emergent generalisation to novel objects and instructions
- OpenVLA is an open-weight 7-billion-parameter model trained on the Open X-Embodiment dataset, a widely used baseline
- SmolVLA is a compact 450-million-parameter model that runs on modest hardware, with asynchronous inference that decouples action prediction from execution
How It Is Trained
VLAs are trained by behavioural cloning on demonstrations, so data collection is the central practical bottleneck. A pretrained VLA is fine-tuned on demonstrations collected on the target robot so that its action space matches that robot and gripper.
What the data looks like:
- synchronised sequences of camera images, robot state, and actions, recorded at a fixed rate
- collected by teleoperation or by scripted expert trajectories
- fine-tuned with parameter-efficient methods when hardware is limited
VLA vs Modular Pipeline
The contrast that the comparison measures:
- the modular pipeline keeps perception, planning, and control as separate inspectable modules, so a failure can be tagged as detection-stage or grasp-stage
- the VLA entangles perception and control in one model, so it has no hard failure boundary, but its behaviour on novel objects is less predictable
- both are run closed-loop, so the comparison is like for like rather than a one-shot script against a continuously correcting policy
At deployment the VLA runs on the UR5E manipulator, where a thin mapping layer reads the current pose, applies the predicted increment, and issues the resulting target to the robot controller.