Overview
- this is the output side: how the LLM tells an image, audio or video generator what to make
- three ways: plain text tokens, continuous embeddings (signal tokens) and codebook tokens
- each way has its own pros and cons, mostly about end-to-end training and what text cannot express
- it ends with how text is generated, and which codebook decoders are used for images and audio
The decoding-side connection is how an MLLM passes its output to a multimodal decoder (a generator for images, audio or video). The LLM itself only produces tokens or hidden states, so it needs a way to tell the decoder what to make. There are three ways to pass this message.
Method 1: text tokens
The LLM writes a normal text response, for example a description of the image to draw. A separate multimodal decoder reads this text and creates the content.

Representative MLLMs: Visual-ChatGPT, HuggingGPT, GPT4Video, MM-REACT, ViperGPT, ModaVerse and Vitron.
- pro: a high performance lower bound, because each decoder is already a strong model
- pro: more efficient, because no tuning is needed
- con: no end-to-end tuning, because a gradient cannot flow back through text
- con: a limited performance upper bound, because some multimodal signals cannot be described well in text
Method 2: continuous embeddings (signal tokens)
The LLM outputs special signal tokens. Instead of turning them into words, their hidden vectors (soft embeddings) go directly to the decoder.

- the system can be tuned end to end, so the instruction reaches the decoder more efficiently
- a vector can carry information that text alone cannot express well, such as the numeration of vision (exactly how many objects to draw) and the visual-spatial relational semantics (where objects are in relation to each other)
Method 3: codebooks
The LLM outputs discrete codebook tokens. These are de-tokenized into multimodal embeddings, which the decoder turns into content.

- it can be tuned end to end for higher efficiency in passing commands
- it expresses multimodal signals that text cannot capture
- it supports autoregressive multimodal token generation, so the LLM generates image or audio tokens one by one exactly as it generates words
Generation of text
An LLM generates text naturally.
- the input text, for example "Paris is the city", is split by a tokenizer into token IDs
- the LLM reads the IDs and outputs logits, one score for every word in the vocabulary
- a softmax turns the logits into probabilities, for example "of" 0.337, "that" 0.153, "where" 0.094
- the next token is chosen from these probabilities, and the process repeats

Generation through codebooks
For other modalities, generation goes through a codebook and a matching decoder.
- visual generators (image and video): VQ-VAE with a codebook, or VQ-GAN with a codebook
- audio generators: SpeechTokenizer with a Residual Vector Quantizer (RVQ), or SoundStream with an RVQ; an RVQ quantizes the signal in several stages, where each stage codes the error left by the stage before
Unified-IO 2 is one example. Its output tokens are split into groups. Text, action, detection and keypoint outputs are decoded by a BPE text decoder. Images, surface normals, segmentation and depth are decoded by a VQ-GAN decoder. Speech, environmental sound and music are decoded by a separate VQ-GAN audio decoder.

Note
Codebooks appear on both sides of an MLLM, for input tokenization and for output generation, see Multimodal Tokenization and Codebooks.