Overview

  • this is the output side: how the LLM tells an image, audio or video generator what to make
  • three ways: plain text tokens, continuous embeddings (signal tokens) and codebook tokens
  • each way has its own pros and cons, mostly about end-to-end training and what text cannot express
  • it ends with how text is generated, and which codebook decoders are used for images and audio

The decoding-side connection is how an MLLM passes its output to a multimodal decoder (a generator for images, audio or video). The LLM itself only produces tokens or hidden states, so it needs a way to tell the decoder what to make. There are three ways to pass this message.

Method 1: text tokens

The LLM writes a normal text response, for example a description of the image to draw. A separate multimodal decoder reads this text and creates the content.

Representative MLLMs: Visual-ChatGPT, HuggingGPT, GPT4Video, MM-REACT, ViperGPT, ModaVerse and Vitron.

  • pro: a high performance lower bound, because each decoder is already a strong model
  • pro: more efficient, because no tuning is needed
  • con: no end-to-end tuning, because a gradient cannot flow back through text
  • con: a limited performance upper bound, because some multimodal signals cannot be described well in text

Method 2: continuous embeddings (signal tokens)

The LLM outputs special signal tokens. Instead of turning them into words, their hidden vectors (soft embeddings) go directly to the decoder.

  • the system can be tuned end to end, so the instruction reaches the decoder more efficiently
  • a vector can carry information that text alone cannot express well, such as the numeration of vision (exactly how many objects to draw) and the visual-spatial relational semantics (where objects are in relation to each other)

Method 3: codebooks

The LLM outputs discrete codebook tokens. These are de-tokenized into multimodal embeddings, which the decoder turns into content.

  • it can be tuned end to end for higher efficiency in passing commands
  • it expresses multimodal signals that text cannot capture
  • it supports autoregressive multimodal token generation, so the LLM generates image or audio tokens one by one exactly as it generates words

Generation of text

An LLM generates text naturally.

  1. the input text, for example "Paris is the city", is split by a tokenizer into token IDs
  2. the LLM reads the IDs and outputs logits, one score for every word in the vocabulary
  3. a softmax turns the logits into probabilities, for example "of" 0.337, "that" 0.153, "where" 0.094
  4. the next token is chosen from these probabilities, and the process repeats

Generation through codebooks

For other modalities, generation goes through a codebook and a matching decoder.

  • visual generators (image and video): VQ-VAE with a codebook, or VQ-GAN with a codebook
  • audio generators: SpeechTokenizer with a Residual Vector Quantizer (RVQ), or SoundStream with an RVQ; an RVQ quantizes the signal in several stages, where each stage codes the error left by the stage before

Unified-IO 2 is one example. Its output tokens are split into groups. Text, action, detection and keypoint outputs are decoded by a BPE text decoder. Images, surface normals, segmentation and depth are decoded by a VQ-GAN decoder. Speech, environmental sound and music are decoded by a separate VQ-GAN audio decoder.

Note

Codebooks appear on both sides of an MLLM, for input tokenization and for output generation, see Multimodal Tokenization and Codebooks.