Overview

  • this note is about turning an image or a sound into discrete tokens, like word IDs, so one LLM can read and write every modality
  • a codebook is a learned lookup table: each feature vector is swapped for the index of its closest row
  • read the shape section first: feature map , codebook , one token per spatial position
  • VQGAN is explained in three stages: train it to rebuild images, train a Transformer on its tokens, then generate new images
  • then three models that use the idea: SEED, AnyGPT and VideoPoet

Multimodal tokenization turns a non-text signal, such as an image or a sound, into discrete tokens, which are integer IDs like the word tokens of a language model. Once every modality is a sequence of tokens, one LLM can read and write all of them with ordinary next-token prediction.

What a codebook is

In general, a codebook describes the contents, structure and layout of a data collection. For example, a survey codebook lists each variable name, its label, the question text and the meaning of each value.

In machine learning, a codebook is a learned table of vectors. Each row has an index. A continuous feature is replaced by the index of the closest row, so the continuous signal becomes a discrete token.

Why not use pixel values as tokens

  • one token per pixel gives far too many tokens: a 256 by 256 image has 65,536 pixels, which is too long a sequence for an LLM
  • a pixel value carries no meaning on its own, it only says how bright one dot is
  • a codebook row is a learned pattern, such as part of an eye, a fur texture or a shadow edge
  • so pixels are like letters, and codebook tokens are like words

Shapes at each step

  • the encoder outputs a feature map of shape , for example
  • is the number of spatial positions, here 256, and each position describes one region of the image
  • is the number of channels (the hidden dimension), so each position holds one 256-channel vector
  • the codebook has shape , for example
  • is the codebook size, the number of rows, and it works like the vocabulary size of an image "language"
  • each row has the same number of channels as one position's vector, so the two can be compared
  • is fixed before training, and only the values inside the rows are learned

VQGAN: learning a codebook for images

VQGAN has four parts: a CNN encoder , a codebook , a CNN decoder and a CNN discriminator . A Transformer is added on top. It is used in three stages:

  • stage 1: train VQGAN to turn an image into tokens and rebuild the same image
  • stage 2: freeze VQGAN and train the Transformer to predict tokens
  • stage 3: let the Transformer write new tokens, and decode them into a new image

Stage 1: train VQGAN to rebuild images

The goal is a good "image to tokens to image" translator.

  1. the encoder turns the image into a feature map of shape
  2. quantization swaps the vector at each spatial position for its nearest codebook row, which gives
  3. the decoder rebuilds the same image from , so no new image is made at this stage
  4. the discriminator judges patches of the rebuilt image as real or fake, which pushes the decoder to make sharp images

The quantization step, done once for every spatial position, is:

where:

  • is the -channel vector that the encoder gives at one spatial position
  • is the codebook, the set of all rows
  • is the row with index , from to
  • is the distance between the encoder vector and that row
  • is the closest row, and its index is the token for that position

Points to keep straight:

  • each spatial position is matched on its own, so a feature map gives 256 tokens, whatever the value of
  • is not a "correct answer", it is simply the closest of the rows
  • the same row can be picked at many positions, wherever similar patches appear
  • the index is the token, a single integer that the Transformer reads and predicts
  • the row vector is what the decoder receives, found by looking up the index in the codebook, so storing only the index loses nothing

How the codebook rows are learned

  • the rows start as random vectors, and no row is given a meaning in advance
  • the encoder, the codebook and the decoder are trained together, so that the decoder can rebuild the original image from the tokens alone
  • the original image is the only ground truth, and the rebuild error is the main training signal

Three loss terms work together:

where:

  • is the original image and is the rebuilt image, so the first term is the rebuild loss
  • is stop-gradient, which treats a value as fixed inside that term
  • the second term is the codebook loss: it moves the chosen row towards the encoder output
  • the third term is the commitment loss: it moves the encoder output towards its chosen row, so the encoder does not keep switching rows
  • is a weight on the commitment loss

Picking the nearest row has no gradient, because it is a hard choice. The straight-through estimator copies the gradient that reaches straight back to , as if the picking step were not there. Some versions drop the codebook loss and update each row as a moving average of the encoder vectors assigned to it, which works like k-means clustering during training.

How a row gets its meaning:

  • over many images, patches that look alike (for example eyes) give similar encoder vectors, so they keep picking the same row
  • that row is pulled towards the average of those vectors, so it becomes a prototype of that patch
  • the decoder learns to draw that patch whenever it sees that row
  • many rows end up as patterns that are hard to name, such as a texture or a colour patch

Stage 2: train a Transformer on the tokens

  • VQGAN is frozen
  • many training images are turned into tokens, and each map of indices is flattened, row by row, into a sequence of 256 integers
  • the Transformer learns to predict the next index from the ones before it, in the same way as GPT predicts the next word

The Transformer models the token sequence as:

where:

  • is the whole sequence of codebook indices for one image
  • is the token at position
  • is all the tokens before position
  • is the probability of the next token given the earlier tokens

Stage 3: generate a new image

  1. the Transformer writes 256 indices one by one, each sampled from its predicted probabilities
  2. each index is looked up in the codebook to get its row vector
  3. the decoder turns those vectors into an image that never existed before

A starting prompt, such as a class label or a text caption, can be given to the Transformer, so that it writes the tokens for "a dog" and not just any image.

Why codebooks are useful

  • they support unified understanding and generation of multimodal signals, because both become next-token prediction in one autoregressive framework
  • they are more common in image synthesis: Parti, Muse and MaskGIT (Muse and MaskGIT predict many tokens in parallel instead of one at a time)
  • representative MLLMs that use them: Gemini, CM3 and VideoPoet

SEED

SEED is an image tokenizer whose tokens are designed to be easy for an LLM to read and write.

  • tokenize: a frozen ViT encoder reads the image, a causal Q-Former with learned queries turns the features into causal embeddings, and a codebook turns them into discrete causal codes
  • the causal embeddings are trained to match text embeddings of the caption with a contrastive loss, so the tokens carry meaning
  • de-tokenize: a reverse Q-Former turns the codes into generation embeddings, and a frozen Stable Diffusion decoder draws the image

AnyGPT

AnyGPT uses one tokenizer per modality, for speech, text, images and music.

  • all tokens go into one LLM, with special start and end tokens such as <sos> and <eos> around each modality block
  • on the output side, a matching de-tokenizer turns speech tokens, image tokens or music tokens back into a real signal

VideoPoet

VideoPoet is an LLM for zero-shot video generation.

  • text is encoded by a T5 encoder
  • images, depth and optical flow, and video are encoded by the MAGVIT-v2 tokenizer into visual tokens
  • audio is encoded by the SoundStream tokenizer into audio tokens
  • all tokens form one sequence, marked by special tokens such as <bov_i> and <eov_i>, and the input part (the prefix) uses bidirectional attention

Note

SEED, AnyGPT and VideoPoet all aim to embed every modality into a joint representation space of language, so that the LLM can treat each modality like a foreign language. The other way to join modalities, through continuous features and a projector, is in Input-side Projection.