RT-2 builds on top of existing large vision-language models, VLM (PaLI-X and PaLM-E), instead of RT-1 where it trained a transformer from scratch on robot demonstration data alone, so it has no knowledge on anything apart from robot data.
RT-2 then teaches this already-knowledgeable model to also output robot actions, by training further on robot demonstration data alongside its original web data.
Input
- A single current image from the robot's camera (not the 6 images like in RT-1)
- A natural language instruction (same as RT-1)
Both image and the instruction are feed into the VLM's existing input pipeline. This is the same pipeline that model would normally use if give it an image and ask "what is in this image", as a visual question-answering task. Nothing robot specific is added to the already existing pipeline yet.
Image goes through the VLM's own vision encoder, instructions through its own text tokenizer. This is the core of RT-2's design choice which is to reuse not rebuild (like in RT-1).
Output
A predicted action, but represented in a different way compared to RT-1. RT-1 had a dedicated action head (neural network) outputting 11 numbers directly. However, RT-2 reuses the VLM's own existing text-generation mechanism.
Recall: action space of 11 numbers (Δx, Δy, Δz, Δroll, Δpitch, Δyaw, gripper, Δx, Δy, Δyaw, mode) all have numbers ranging from 0-256 bins. These contain values such as +0.039m, -0.018m.
For PaLI-X:
- PaLI-X's vocabulary already contains unique tokens for integers up to 1000. So bin178 just directly reuses the existing token for the number "178"
For PaLM-E:
- PaLM-E's vocabulary doesn't have that convenient spare capacity
- So we take the 256 least frequently used tokens in PaLM-E's existing vocabulary, and overwrote the meaning
- Token ID that used to mean some rare and unused word now means "action bin 178"
- The model has to learn this new meaning during co-fine-tuning
What this gives is a full generated action from the generated tokens:
Generated tokens: 178 122 130 128 129 127 1 128 128 128 0
↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓
Meaning: Δx Δy Δz Δroll Δpitch Δyaw grip bx by byaw mode
This is identical in shape to the RT-1 where there are 11 bin indices. The difference is just how they where produced. Last step is the de-tokenizer which is similar to the RT-1. This is to convert the 11 bin indices back to physical value, similar to the RT-1's lookup table.
However we notice that the decoder can realistically generate anything thing they want, even generate gibberish that are not action tokens. This where we use Constrained Decoding.
Constrained Decoding
During inference, generation at action-token positions is masked so only the reserved 256 action-bin tokens are eligible. This is enforced precisely because the token IDs were deliberately reserved beforehand.
Co-Fine-Tuning
The risk: if we fine-tune a VLM purely on robot trajectory data, it tends to degrade or forget its original web-scale knowledge which is the exact thing we wanted to keep. Pure fine-tuning risks collapsing performance back toward something like RT-1.
The fix: train on a mixture of robot trajectory data and the original web-scale vision-language data (captioning, visual question answering) simultaneously, in the same training run, not as sequential stages. For RT-2-PaLM-E, robot data makes up about 66% of the training mixture, with the remainder staying web-scale data. This is why the paper calls it "co-fine-tuning" specifically, not just "fine-tuning" since the model is pulled toward action generation and general visual-language competence at the same time.
Two Architectures
The paper reports two architectures: PaLM-E (12B parameters) and PaLI-X (55B parameters), both adapted using the same co-fine-tuning recipe.
PaLI-X-55B performed stronger across most evaluation axes while PaLM-E had a relative edge on math-reasoning tasks. Larger backbones generalized better which is the scaling trend, not a single fixed design.
Thinking Process
Since the backbone is capable, RT-2 can be trained on "instruction, plan, and action" instead of purely "instruction, action". This allows the model to generate the plan first, which is free text and unconstrained generation, and subsequently action tokens which are constrained (only 256 valid tokens allowed). How the model "knows" which to generate tokens is via training where each example is labelled as instruction --> plan --> action tokens. The model learns from the training data's structure that after generating a plan, the next tokens should produce action tokens. The constrained decoding mask is applied by the inference code specifically at that later position in the sequence.
Results
For tasks seen in training, RT-2 performed similar to RT-1. But on novel objects, backgrounds, and environments, it achieved roughly 2x higher success rates compared to RT-1 and other baselines.
This worked better because the underlying VLM already had web pretraining which allowed it to perform in novel object handling, numerical/symbolic reasoning, human recognition. RT-1 had nothing like that since its knowledge is only fully bounded by what it saw during training.
| RT-1 | RT-2 | |
|---|---|---|
| Backbone | Trained from scratch | Pretrained VLM (PaLI-X / PaLM-E) |
| Visual input | 6-frame history | Single current image |
| Action output | Dedicated action head | Reused text-generation vocabulary |
| Knowledge source | Robot data only | Robot data + web-scale vision-language data |
| Generalisation | Bounded by training distribution | Extends via web pretraining (2x on novel objects) |
| Reasoning | None | Optional chain-of-thought (plan before action) |