Overview

  • these models take text plus one other kind of input, and answer only in text
  • they all use one recipe: an encoder makes features, and the LLM answers the question about them
  • the note goes through four groups (image, video, 3D and audio), each with a model list and a few models in detail
  • image models (BLIP-2, LLaVA, Flamingo, MiniGPT-4) are the most studied

A multimodal perception MLLM takes text plus one other modality as input and gives a text response. It understands the image, video, 3D scene or audio, but it does not generate that modality. All of them follow the same recipe from MLLM Architecture II (Encoder-LLM-Decoder): an external encoder turns the signal into features that the LLM can understand, and the LLM answers the text instruction about it.

Modalities

An MLLM can in principle perceive language, images, sound, video, depth and 3D, spectrograms, time series, graphs, code, documents and tables, infrared and radar, touch, smell and heat maps. The most studied pair is language plus vision.

Image-perceiving MLLM

  • input: text and an image; output: text
  • an external image encoder produces an LLM-understandable visual feature, which is fed into the LLM
  • the LLM interprets the image according to the text instruction and produces a text response

Models: Flamingo, Kosmos-1, BLIP-2, mPLUG-Owl, MiniGPT-4, LLaVA, InstructBLIP, Otter, VPGTrans, Chameleon, Qwen-VL, GPT-4V and SPHINX.

  • BLIP-2: a frozen image encoder and a Q-Former with learnable queries; it is trained in two stages, first vision-language representation learning, then vision-to-language generative learning with a frozen LLM
  • LLaVA: a vision encoder, then a projection matrix , then the language model; the projected image tokens and the instruction tokens go into the LLM together
  • Flamingo: frozen vision encoders, Perceiver Resamplers, and new gated cross-attention layers placed between the frozen LLM blocks; it accepts interleaved images and text
  • MiniGPT-4: a frozen ViT and Q-Former, then one linear layer, then the Vicuna LLM

Video-perceiving MLLM

  • an external video encoder turns the video frames into an LLM-understandable feature
  • the LLM interprets the video according to the text instruction and answers in text

Models: VideoChat, Video-ChatGPT, Video-LLaMA, PandaGPT, MovieChat, Video-LLaVA, LLaMA-VID and Momentor.

  • Video-ChatGPT: a frozen pretrained CLIP encoder reads each frame; the features are averaged over time (temporal pooling) and over space (spatial pooling); a linear layer maps them into the Vicuna LLM
  • Video-LLaVA: an encoder from LanguageBind reads both images and videos, and a shared projection maps both into Vicuna, so one model learns images and videos together

3D-perceiving MLLM

  • an external encoder turns 3D information (scenes or point clouds) into an LLM-understandable 3D feature
  • the LLM interprets the 3D input according to the text instruction and answers in text

Models: 3D-LLM, 3D-GPT, LL3DA, SpatialVLM, PointLLM and Point-Bind.

  • 3D-LLM: builds 3D features from many 2D views of a scene (by direct reconstruction, gradSLAM or a neural field), passes them through a perceiver to the LLM, and can answer with location tokens, for example where the chair is
  • PointLLM: a point-cloud encoder and a projector turn a point cloud into tokens that sit next to the text tokens in the LLM

Audio-perceiving MLLM

  • an external audio encoder turns sound, speech or music into LLM-understandable features
  • the LLM interprets the audio according to the text instruction and answers in text

Models: SpeechGPT, VIOLA, AudioPaLM, SALMONN and MU-LLaMA.

  • SpeechGPT: a discrete speech-unit extractor turns speech into unit tokens; the LLM is trained on cross-modal instruction data and on chain-of-modality instructions; a unit vocoder can turn output units back into speech
  • SALMONN: a Whisper encoder (for speech) and a BEATs encoder (for general audio) are joined by a window-level Q-Former, and the LLM is adapted with LoRA