RT-1

A Transformer-based model that was trained on a large dataset of multi-task demonstrations and showcase how it generalizes to new tasks, how it is robust to changes in its environment and how it allows to execute long-horizon instructions.

Input and Output

We treat RT-1 as a function. Input goes in, passes through the function, resulting in particular output that comes out.

Input

A group of 6 RGB frames from a fixed camera watching the workspace. This is so that the model can see a window of recent history of its motion and context.
A natural language instruction that contains just 1 sentence which lasts throughout every RGB frame in an episode.

Output

A single action vector of what the robot should do at that particular instance, at that time step. It contains 11 numbers:

7 for the arm which is the change in (Δ) TCP pose (Δx, Δy, Δz, Δroll, Δpitch, Δyaw, gripper)
3 for the mobile base movements (Δx, Δy, Δyaw). RT-1 was trained on a mobile manipulator so that base can move around.
1 mode switch which is a discrete flag, that maps to either "control the arm", "control the base", or "terminate episode"

Design choice of deltas instead of absolute makes sense because the robot doesn't need to know what exists in the world. It just outputs small corrections of actions based on what it sees, making it naturally closed loop.

Also, the 11 numbers aren't floats. They are bucketed into 256 bins, and the model picks a bin. Meaning, we 'hardcode' the discrete bins, such as ..., -0.10, -0.05, -0.00, 0.05, 0.10, .... This allows the model to train better. Regression heads (action head part) tend to predict mean, so if half demos went left and half demos went right, output 0. But classification heads predict a distribution (60% chances is left, 40% chances is right) and then samples. Action now becomes "tokens", just like language tokens.

6 Stages

[6 images + instruction string]
              │
              ▼
   (1) Image tokenizer    ← FiLM-EfficientNet-B3
              │  ~486 tokens
              ▼
   (2) TokenLearner       ← learned compression
              │  48 tokens
              ▼
   (3) Transformer        ← 8-layer decoder
              │  11 action tokens
              ▼
   (4) Action de-tokenizer  ← bins → motor commands
              │
              ▼
   [Δx, Δy, Δz, Δroll, Δpitch, Δyaw, gripper, base...]

Estimated 35M parameters total.

Stage 1: Tokenize

Input: the 6 images + the instruction
EfficientNet-B3: this is the vision module that turns each image into a grid of feature vectors, each input image is 300 by 300 pixels, and comes out as a 9 by 9 grid of 512-dim vectors, (9 , 9, 512), where each of the 512 vectors summarizes one patch of the image
Output: flat sequence of "tokens" (vectors) the transformer can chew on. To be precise, 6 images x 81 patches = 486 vision tokens.
FiLM: FiLM allows the CNN to have context on the instructions as it processes the image. Let's say we wanna pick up the coke can. There are 100+ cans in the image, but only 1 coke can. Without FiLM, CNN will just encode everything, which wastes computation.

How does FiLM work:

Encode the instruction with a small language model (Universal Sentence Encoder), getting one 512-dim embedding for the whole sentence. This language model (internally a Transformer) just takes in the natural language input and outputs vectors.

Pass this embedding through a tiny network to produce 2 vectors: γ (multiplier) and β (offset)

At each layer of the CNN, the channels are modified by γ and β. γ and β are each vectors of length 512. Meaning they have an impact on every 512 channels. See implementation below:

for each channel c (0 to 511):
    for each spatial position (i, j):
        features[i, j, c] = γ[c] * features[i, j, c] + β[c]

Explanation for γ and β and how it affects the CNN channels:

All 81 (9 x 9) spatial positions of a single channel (out of total 512), gets the same γ and β.
Let's say we wanna grab a red ball. If a particular channel should be given more attention (maybe because this particular channel is a red colour detector), then the γ will multiply it, giving it more activation.
β differs from it since it is an offset. If a channel's activation is high but its irrelevant (maybe a channel that identifies sharp features, straight lines, sharp corners), then this is where β can just turn this channel off completely by providing that offset.
Another case if let's say we wanna grab something from the drawer, but while moving there it is not open. Channels that identify the drawer handle's features might have low activation since its not an opened drawer which is what we would expect. However, β is able to raise the entire channel's baseline, so even weak/ambiguous signals start to register.

Stage 2: Token Learner

Input: 486 tokens (6 images x 81 patches) which is too much for transformers because its costs scales quadratically with sequence length
To fix this we use a Token Learner

How it works:

a small network looks at all 486 tokens
it then produces 486 attention weights (on per input token)
the slot's output = weighted sum of all 486 tokens, using those weights produced

The attention weights are learned.During training, the network figures out which tokens each slot should attend to. After training, slot 1 might naturally specialize in "task-relevant object location", and slot 2 in "gripper position", slot 3 in "obstacles", etc.

This is done for every image. Meaning from 6 images * 81 tokens = 486 raw tokens, it now becomes 6 images * 8 slots = 48 summary tokens. Therefore, these 48 tokens go into the transformer.

Stage 3: Standard Transformer

48 tokens come from the TokenLearner in an orderly manner:

[frame1_slot1, frame1_slot2, ..., frame1_slot8,    ← 8 tokens from oldest frame
 frame2_slot1, ..., frame2_slot8,                  ← 8 tokens from next frame
 ...
 frame6_slot1, ..., frame6_slot8]                  ← 8 tokens from newest frame

Plus a positional encoding added to each token, allowing the model to understand about temporal ordering and can reason about change over time. This positional encoding encodes time instead of word order (like in language transformers).

This is also a decoder only transformer which is one that uses causal attention. This is where each token can only attend to itself and earlier tokens, not later ones. In a language model, this is what lets it predict the next word. In RT-1, this means that later frames can attend to earlier frames, but not the other way around. Causal attention enforces the correct direction of information flow, where the action of the robot depends on what it is doing right now.

After 8 layers of self-attention and feed-forward, the transfomers produces 11 action numbers (the 7 arm dims, 3 base, and 1 mode flag). So the RT-1 generates these 11 probability distributions over 256 bins autoregressively, same as how a language model generates words. RT-1 predicts action dim 1 (Δx), then conditions on it to predict dim 2 (Δy), then dim 3 (Δz), and so on, until all 11 are done. It is autoregressive because action dimensions are correlated. If we move in positive x direction by 5 cm, the appropriate amount to move in y and z are dependent on how much i move in x. This output head that is supposed to provide the 11 action numbers, actually runs 11 times. The output head is essentially a small network on top that converts the hidden state into a bin choice.

Here is the loop:

hidden = transformer_body(48 vision tokens)   ← runs ONCE, expensive
 
action_so_far = []
for i in range(11):
    # Look at hidden state + actions chosen so far
    bin_choice = output_head(hidden, action_so_far)   ← cheap, runs 11x
    action_so_far.append(bin_choice)
 
return action_so_far   # 11 bin indices

Stage 4: Action De-Tokenizer

The transformer gave 11 bin indices: [178, 122, 130, 128, 129, 127, 1, 128, 128, 128, 0], these are integers from 0 to 255 which is useless for the robot. We pass this through a de-tokenizer which is like a lookup table that convert each bin index back to a physical value.

Δx:  bin 178 → +0.039 m   (push right)
Δy:  bin 122 → -0.005 m   (tiny nudge back)
Δz:  bin 130 → +0.008 m   (slight lift)
Δroll, Δpitch, Δyaw: small rotations
gripper: bin 1 → close
base dims: bin 128 → no motion
mode: bin 0 → "control arm"

Its essentially just bin_index × bin_width + bin_min. The output goes straight to the robot controller as a delta command, applied to the current TCP pose. Next timestep, new images come in, the whole pipeline runs again.

This leads to the next iteration of RT-2.

JJ

Explorer