Overview

  • a unified MLLM can both understand and generate, so it can answer with an image or audio, not only text
  • the note moves from one extra modality (image, audio) to several at once, then to any-to-any
  • models in detail: GILL, EMU, Chat-UniVi, Macaw-LLM and NExT-GPT
  • NExT-GPT is the main any-to-any example, and only its small projection layers are trained

A unified MLLM does both perception and generation. After the central LLM understands the meaning of the input, it can produce text and also another modality, such as an image or audio. This goes one step beyond Multimodal Perception MLLMs, which only answer in text.

Image and video

  • input: text and images; output: text and images
  • models: GILL, EMU, MiniGPT-5, DreamLLM, LLaVA-Plus and LaVIT

  • GILL: the LLM outputs special image tokens such as [IMG1]; a small GILLMapper network maps their output embeddings into the text-encoder space of Stable Diffusion, which draws the image; the same embeddings are also used to retrieve real images (trained with an InfoNCE loss), while generation is trained with an MSE loss
  • EMU: an EVA-CLIP encoder and a causal Transformer turn images into embeddings inside the text sequence; the LLaMA model predicts both the next word (classification) and the next visual embedding (regression); a Stable Diffusion decoder turns the predicted embeddings into an image

Audio

  • input: text and audio; output: text and audio
  • models: AudioGPT, SpeechGPT, VIOLA and AudioPaLM
  • SpeechGPT works on discrete speech units, so it can listen and speak in one model
  • AudioGPT uses the LLM to analyse the task, assign the right audio model (such as Whisper, Make-An-Audio, VISinger or DiffSinger), and generate the response, so it is closer to the scheduler design

Several modalities at once

In reality, modalities are often strongly connected at the same moment. A video has sound, and an image has a caption. So an MLLM often needs to understand several non-text modalities at once, not only one. Common scenarios are image plus video, audio plus video, image plus video plus audio, and any-to-any.

Text, image and video

  • input: text, images and videos; output: text (sometimes also images and videos)
  • models: Video-LLaVA, Chat-UniVi and LLaMA-VID

Chat-UniVi represents images and videos with the same kind of dynamic visual tokens. Similar neighbouring tokens are merged step by step (spatial merging), and video frames are grouped into events (temporal merging). Thus one image or one event needs only a few tokens, and the same LLM handles both.

Text, image, video and audio

  • input: text, audio, images and videos; output: text (sometimes also the other modalities)
  • models: PandaGPT, Video-LLaMA, AnyMAL, Macaw-LLM, VideoPoet, ImageBind-LLM, LLMBind and LLaMA-Adapter

Macaw-LLM has three modules:

  • modality module: an image encoder, an audio encoder and a video encoder
  • alignment module: maps all the encoded features into the LLM embedding space
  • cognitive module: the LLM, which reads the image, audio and video tokens with the instruction and writes the response

Any-to-any

An any-to-any MLLM takes text, audio, images and videos as input and can freely generate text, audio, images, videos, or a combination.

Models: NExT-GPT, Unified-IO 2 (without video), AnyGPT (without video), CoDi-2 and ModaVerse.

NExT-GPT has five stages from left to right:

  1. multimodal input encoding: frozen image, audio and video encoders
  2. LLM-centric alignment: trainable input projections map each encoder output into the LLM space
  3. LLM-based semantic understanding: a frozen LLM reasons over all inputs
  4. instruction-following alignment: trainable output projections map the LLM output to each generator
  5. multimodal output generation: frozen image, audio and video diffusion models create the outputs

Only the small projection layers are trained, so the system is cheap to build and new modalities can be added.