1 Overall Architecture

1.1 Introduction to SmolVLA

SmolVLA is essentially two models combined into one.

a small frozen-ish VLM that only perceives
a seperate ~100M action expert that acts via flow matching

Total of around 450M parameters, roughly 15x smaller, and actions come out as a continuous chunk in one forward pass instead of token by token like in other VLA models.

1.2 Component A : Perception (VLM)

1.2.1 Base Model

SmolVLM-2 consists of both SigLIP vision encoder & SmolLM2 language decoder. It is multi-image/video capable, which is useful when we have both wrist strap and top cameras.

1.2.2 Token Construction

There are 3 streams concatenated into one sequence for the decoder:

Visual Tokens
- 64 tokens per camera frame
- image resized to a fixed square (512 x 512 in SmolVLA's config)
- SigLIP has a patch size of 16
- Meaning 32 x 32 = 1024 patch embeddings (total no. of patches) from 1 single image
- Each patch embeddings have dimension (channels) of 768
- Currently we have [32 x 32 x 768] (where it represents grid and no. of dims)
- This is passed into Pixel Shuffle
- What this does is we take 16 neighbouring patches that form a 4x4 block and concatenates them together to form a vector
- Since originally we have 1024 patches, now we will have 64 non-overlapping blocks of 16 patches each
- This reduces to [8 x 8 x 12288]
- 12288 is the result of 768 initial channels per patch x 16 patches per block
- We then flatten it to [64, 12288] (64 tokens)
- Next we pass it through a MLP projector that maps 12288 to 960
- Final visual output per camera: [64, 960]
Text Tokens
- Uses the standard SmolLM2 Byte Pair Encoding (BPE) tokenizer
- BPE's tokenizing works by giving frequent chunks of text single tokens, whereas rare words fall back to smaller pieces
- Unlike OpenVLA where we override the 256 least-frequently-used tokens in the Llama tokenizer and reassigns them to action bins
- SmolVLA is different when text goes in, the vocab is untouched
- This is because actions never live in token space at all, they are produced downstream by the flow-matching expert as continuous floats
- Each token ID is mapped through the embedding table to a 960-d vector
- Output is [n_text * 960]
State Tokens
- The robot reports its proprioceptive state at time * $t$
- That is ~6 floats (5 joint angles + gripper), represented as $s_{t} \in R^{6}$
- This state vector that has a shape of $(6, 1)$ undergoes matrix multiplication with a learnable weight parameter, $W$ with addition of learnable bias parameter, $b$ also of 960-parameters
- This is representated as: $z = W s_{t} + b, w h er e : W \in R^{960 * 6}, b \in R^{960}$
- Each of the 960 output dimensions is a learned weighted sum of the 6 input numbers + a bias
Combination of Tokens
- [64 img tokens | 64 img tokens | 12 text tokens | 1 state token] are all 960 dimensions
- All streams concatenated along the sequence (row) axis
- The hidden axis (column) already match at 960 for every modality
- example, 64 (image 1 from gripper) + 64 (image 2 from top) + 12 (text) + 1 (current state) = 141 --> therefore, [141 x 960]

1.2.3 Visual Token Reduction

Mechanism already described above in 1.2.2 Visual Token. But this part adds to it:

SmolVLM-2 normally uses image tiling (multiple crops + global image --> hundreds of tokens)
However, SmolVLA drops tiling (as described above)
it keeps only the global image and capped at 64 tokens/frame

1.2.4 Layer Skipping

SmolLM2 is a stack of lets say L = 32 transformer layers in sequence
so what happens is our input matrix of previously mentioned [141 960] goes through each transformer layer and outputs the same dimension of features: [141 960]
However, the action expert does not read the VLM's last-layer features
It reads features only up to layer N = L/2
Meaning if our SmolLM2 has 32 transformer layers, then:

input [141 × 960]
   │
 Layer 1   → hidden state h₁  [141 × 960]
   │
 Layer 2   → hidden state h₂  [141 × 960]
   │
  ...
 Layer 16  → hidden state h₁₆ [141 × 960]   ← N = L/2
   │
  ...
 Layer 32  → hidden state h₃₂ [141 × 960]   ← "last layer features"

for control tasks, the most useful features are mid-stack
final layers are specialized for next-token prediction, which SmolVLA isn't doing
this halves the cost of both the LLM and the expert's cross attention
OpenVLA needs the full stack beause its LLM head in the action head

1.3 Component B: The Action Expert

This is the half that actually produces motion. 2 key points here:

how it generates actions (flow matching)
how it's wired to the VLM (interleaved attention)

1.3.1 What it is

A seperate small transformer written $v_{θ}$ ~100M params, with a hidden dim is about 0.75x the VLM's. Its input is the VLM features $o_{t}$ + a noised action chunk. Its output is a clean chunk of continuous actions $A_{t} = (a_{t}, ..., a_{t + n})$ . Typically ~50 future timesteps. all in one forward pass.

1.3.2 Flow Matching: The generation Method

Flow matching is just a training method, a way of training the model. The model that is being trained here in this action head is another transformer. This replaces OpenVLA's classify into action bins method. This idea is borrowed from diffusion style generative models.

How it works in simple terms is that this action expert, which is essentially another model, takes in both VLM features + pure random noise as inputs. It will then repeatedly reduce the noise (called integration steps) which results in an output of an action chunk.

During Training, we take a ground truth action chunk $A_{t}$ from a demonstration. Corrupt it with noise to a level $τ$ :

A_{t}^{τ} = τ A_{t} + (1 - τ) ϵ, ϵ \sim N (0, I)

$A_{t}^{τ}$ is the action chunk being denoised
$τ$ is the noise level
$A_{t}$ is the clean target/label, which is only used to build the noisy version and to define the true velocity it should predict

At $τ = 0$ , its pure noise. At 1, its the clean action. Now we ask the action expert, given this noisy chunk and the VLM features, which direction points back towards the clean chunk? That direction is velocity. The expert is trained to predict it:

L^{τ} (θ) = E [∥ v_{θ} (A_{t}^{τ}, o_{t}) - u (A_{t}^{τ} ∣ A_{t}) ∥^{2}]

where $u$ is the true velocity (the target direction) and $v_{θ}$ is the expert's guess, which takes in 2 arguments, namely the noisy, and $o_{t}$ are the features from the VLM. It's just a regression on vectors. The MSE between predicted and true velocity.

Important: During training, the denoise happens in one-shot, not iteratively like during inference.

pick one random noise level $τ$
build one blended point
expert makes one prediction
compute loss
repeat the whole process again... no iteratively denoise

During Inference, we start from giving the action expert pure noise $A_{t}^{0} \sim N (0, I)$ and the VLM features. Repeatedly ask the expert for the velocity and step in that direction a handful of times (~10 integration steps), until we arrive at the clean action chunk. There is no autoregression since the whole chunk is denoised together.

1.3.3. Interleaved cross and self attention

What happens inside this transformer is that the action chunks (initially noisy) flows through the whole transformer network while the VLM features is only used as reference. We use Cross Attention Layer as a way for the action chunks in their particular path (or flow in the transformer architecture) to stil access and have knowledge about the VLM features. We then use Self Attention Layer so that within this action chunk path that flows through this transformer, the action 2 is a sensible option after making action 1. So that each actions understands each other.

This is SmolVLA's genuine architectural novelty vs $π_{0}$ . The expert's layers alternate 2 types:

Cross Attention Layer: action tokens (the noisy chunks) cross-attend to the VLM features. This is how perception enters the expert. It's one directional meaning actions read perception, but perception never reads the actions.
Self Attention Layer: action tokens attend to each other. This enforces temporal cohenrence within the chunk where an action at t + 5 is coherent with action at t + 4, meaning its causal

They alternate CA, SA, CA, SA, .. which is:

Cheaper: Since the VLM features never enter the expert's self attention QKV, we avoid attention over the full (perception + action) sequence at every layer
Empirically Better: the paper's ablation shows interleaving beats CA-only or SA-only

NEXT PARTS ARE GENERATED!

2 Asynchronous Inference: the other real novelty

This is SmolVLA's second headline contribution (alongside the architecture). It's about deployment, not the network.

2.1 The problem

The model outputs a chunk of ~50 actions. Naive approach: execute all 50, stop, run the model again, execute next 50. The robot freezes during each inference. This causes jerky motion, wasted time, low effective control rate.

2.2 The fix: decouple prediction from execution

Split into two processes:

Robot client: just executes actions from a queue, in real time
Policy server: runs the VLM + expert to produce new chunks (can even be on a different machine/GPU)

2.3 How it flows

The client executes the current chunk while the server is already computing the next one. When the queue drops below a threshold, the client sends the latest observation and requests a fresh chunk; the server returns it before the queue empties. New chunks overlap/replace the tail of the old queue. Result: the robot never idles, control rate goes up, motion stays smooth. This is why SmolVLA feels responsive despite chunked generation.

3 Training data: the "community-driven" part

3.1 Source

Instead of expensive academic/industrial datasets, SmolVLA is pretrained on ~450–500 community datasets collected on cheap SO-100/SO-101 arms and shared on the LeRobot Hub. This is the affordable-robotics angle in the title.

3.2 The messiness problem

Community data is inconsistent: different camera names, missing or junk task labels, varied conventions. Two cleanup steps make it usable:

Task re-annotation: an off-the-shelf VLM generates a clean, concise instruction for each episode (fixes empty/garbage labels)
Camera-view standardization: heterogeneous camera names mapped to canonical slots (e.g. top / wrist)

3.3 Why it matters

This is the ideological difference from OpenVLA/π₀: those lean on curated large-scale robot data; SmolVLA shows you can pretrain a capable VLA on freely available hobbyist data, on a single GPU.

4 Efficiency recap (already covered, one place)

The four levers that make 450M viable, so they're together in your notes:

Small pretrained VLM (SmolVLM-2) instead of a 7B backbone
64 visual tokens/frame (pixel shuffle, no tiling), giving a 16× shorter visual sequence
Layer skipping at N = L/2: the top half of the LLM never runs
Interleaved CA/SA expert: cheaper than joint attention over perception + action

Plus flow matching for continuous chunked actions (no tokenizer, no autoregression) and async inference for responsiveness.

5 Results & ablations: what actually mattered

5.1 Headline

SmolVLA (~450M) reaches performance comparable to VLAs ~10× larger (e.g. π₀ at billions of params) across simulated (LIBERO, Meta-World) and real SO-100 tasks. It is trainable on one GPU, deployable on CPU/consumer GPU.

5.2 Key ablation findings

(these validate the design choices you learned):

Stopping at L/2 is as good as the full stack, which confirms the top layers aren't needed once the action head is decoupled
Interleaved CA/SA beats self-attention-only, which validates the alternating design
Early/mid LLM layers give better control features than the final layer

6 Consolidated comparison

Axis	OpenVLA	SmolVLA
Params	7B	0.45B
Backbone	large VLM (Prismatic/Llama)	SmolVLM-2 (small)
VLM depth used	all layers	first L/2
Visual tokens	~256+ (tiling)	64 (no tiling)
Proprioception	none	1 state token
Actions	discrete tokens, autoregressive	continuous chunk, flow matching
Action head	the LM head itself	separate Transformer expert
Tokenizer	256 tokens hijacked for action bins	untouched, text only
Inference	synchronous, one action at a time	async, chunked, overlapped
Training data	curated OXE	community SO-100 data

JJ

Explorer

SmolVLA