Overview
- this note is about turning an image or a sound into discrete tokens, like word IDs, so one LLM can read and write every modality
- a codebook is a learned lookup table: each feature vector is swapped for the index of its closest row
- read the shape section first: feature map , codebook , one token per spatial position
- VQGAN is explained in three stages: train it to rebuild images, train a Transformer on its tokens, then generate new images
- then three models that use the idea: SEED, AnyGPT and VideoPoet
Multimodal tokenization turns a non-text signal, such as an image or a sound, into discrete tokens, which are integer IDs like the word tokens of a language model. Once every modality is a sequence of tokens, one LLM can read and write all of them with ordinary next-token prediction.
What a codebook is
In general, a codebook describes the contents, structure and layout of a data collection. For example, a survey codebook lists each variable name, its label, the question text and the meaning of each value.
In machine learning, a codebook is a learned table of vectors. Each row has an index. A continuous feature is replaced by the index of the closest row, so the continuous signal becomes a discrete token.
Why not use pixel values as tokens
- one token per pixel gives far too many tokens: a 256 by 256 image has 65,536 pixels, which is too long a sequence for an LLM
- a pixel value carries no meaning on its own, it only says how bright one dot is
- a codebook row is a learned pattern, such as part of an eye, a fur texture or a shadow edge
- so pixels are like letters, and codebook tokens are like words
Shapes at each step
- the encoder outputs a feature map of shape , for example
- is the number of spatial positions, here 256, and each position describes one region of the image
- is the number of channels (the hidden dimension), so each position holds one 256-channel vector
- the codebook has shape , for example
- is the codebook size, the number of rows, and it works like the vocabulary size of an image "language"
- each row has the same number of channels as one position's vector, so the two can be compared
- is fixed before training, and only the values inside the rows are learned
VQGAN: learning a codebook for images

VQGAN has four parts: a CNN encoder , a codebook , a CNN decoder and a CNN discriminator . A Transformer is added on top. It is used in three stages:
- stage 1: train VQGAN to turn an image into tokens and rebuild the same image
- stage 2: freeze VQGAN and train the Transformer to predict tokens
- stage 3: let the Transformer write new tokens, and decode them into a new image
Stage 1: train VQGAN to rebuild images
The goal is a good "image to tokens to image" translator.
- the encoder turns the image into a feature map of shape
- quantization swaps the vector at each spatial position for its nearest codebook row, which gives
- the decoder rebuilds the same image from , so no new image is made at this stage
- the discriminator judges patches of the rebuilt image as real or fake, which pushes the decoder to make sharp images
The quantization step, done once for every spatial position, is:
where:
- is the -channel vector that the encoder gives at one spatial position
- is the codebook, the set of all rows
- is the row with index , from to
- is the distance between the encoder vector and that row
- is the closest row, and its index is the token for that position
Points to keep straight:
- each spatial position is matched on its own, so a feature map gives 256 tokens, whatever the value of
- is not a "correct answer", it is simply the closest of the rows
- the same row can be picked at many positions, wherever similar patches appear
- the index is the token, a single integer that the Transformer reads and predicts
- the row vector is what the decoder receives, found by looking up the index in the codebook, so storing only the index loses nothing
How the codebook rows are learned
- the rows start as random vectors, and no row is given a meaning in advance
- the encoder, the codebook and the decoder are trained together, so that the decoder can rebuild the original image from the tokens alone
- the original image is the only ground truth, and the rebuild error is the main training signal
Three loss terms work together:
where:
- is the original image and is the rebuilt image, so the first term is the rebuild loss
- is stop-gradient, which treats a value as fixed inside that term
- the second term is the codebook loss: it moves the chosen row towards the encoder output
- the third term is the commitment loss: it moves the encoder output towards its chosen row, so the encoder does not keep switching rows
- is a weight on the commitment loss
Picking the nearest row has no gradient, because it is a hard choice. The straight-through estimator copies the gradient that reaches straight back to , as if the picking step were not there. Some versions drop the codebook loss and update each row as a moving average of the encoder vectors assigned to it, which works like k-means clustering during training.
How a row gets its meaning:
- over many images, patches that look alike (for example eyes) give similar encoder vectors, so they keep picking the same row
- that row is pulled towards the average of those vectors, so it becomes a prototype of that patch
- the decoder learns to draw that patch whenever it sees that row
- many rows end up as patterns that are hard to name, such as a texture or a colour patch
Stage 2: train a Transformer on the tokens
- VQGAN is frozen
- many training images are turned into tokens, and each map of indices is flattened, row by row, into a sequence of 256 integers
- the Transformer learns to predict the next index from the ones before it, in the same way as GPT predicts the next word
The Transformer models the token sequence as:
where:
- is the whole sequence of codebook indices for one image
- is the token at position
- is all the tokens before position
- is the probability of the next token given the earlier tokens
Stage 3: generate a new image
- the Transformer writes 256 indices one by one, each sampled from its predicted probabilities
- each index is looked up in the codebook to get its row vector
- the decoder turns those vectors into an image that never existed before
A starting prompt, such as a class label or a text caption, can be given to the Transformer, so that it writes the tokens for "a dog" and not just any image.
Why codebooks are useful
- they support unified understanding and generation of multimodal signals, because both become next-token prediction in one autoregressive framework
- they are more common in image synthesis: Parti, Muse and MaskGIT (Muse and MaskGIT predict many tokens in parallel instead of one at a time)
- representative MLLMs that use them: Gemini, CM3 and VideoPoet
SEED
SEED is an image tokenizer whose tokens are designed to be easy for an LLM to read and write.
- tokenize: a frozen ViT encoder reads the image, a causal Q-Former with learned queries turns the features into causal embeddings, and a codebook turns them into discrete causal codes
- the causal embeddings are trained to match text embeddings of the caption with a contrastive loss, so the tokens carry meaning
- de-tokenize: a reverse Q-Former turns the codes into generation embeddings, and a frozen Stable Diffusion decoder draws the image

AnyGPT
AnyGPT uses one tokenizer per modality, for speech, text, images and music.
- all tokens go into one LLM, with special start and end tokens such as
<sos>and<eos>around each modality block - on the output side, a matching de-tokenizer turns speech tokens, image tokens or music tokens back into a real signal

VideoPoet
VideoPoet is an LLM for zero-shot video generation.
- text is encoded by a T5 encoder
- images, depth and optical flow, and video are encoded by the MAGVIT-v2 tokenizer into visual tokens
- audio is encoded by the SoundStream tokenizer into audio tokens
- all tokens form one sequence, marked by special tokens such as
<bov_i>and<eov_i>, and the input part (the prefix) uses bidirectional attention

Note
SEED, AnyGPT and VideoPoet all aim to embed every modality into a joint representation space of language, so that the LLM can treat each modality like a foreign language. The other way to join modalities, through continuous features and a projector, is in Input-side Projection.