Overview

  • here the LLM takes in the image or audio features directly, so it is one part of a joined system
  • shape: encoder, then LLM, then decoder, with a small connector between the encoder and the LLM
  • this is the main design today (over 90% of MLLMs), because it has a higher upper bound and can be trained as one model
  • read it side by side with Architecture I, since the two are often compared

In Architecture II, the LLM is a joint part of the system. It receives multimodal information directly from outside, understands it by itself, and then passes instructions to decoders or generators that produce non-text outputs. The structure is Encoder, LLM, Decoder.

Three parts

  • encoder side: text goes straight into the LLM, while each other signal (image, audio, video) first goes through a multimodal encoder that turns it into feature vectors
  • core: the LLM reads the text and the encoded features together, and reasons about them
  • decoder side: text comes straight out of the LLM, while a multimodal decoder turns the LLM output into an image, audio or video when one is needed

Components in detail

A typical model of this type has four trainable or frozen blocks:

  • modality encoder: a pretrained network, such as a vision encoder, that turns raw input into feature vectors, see Multimodal Encoders
  • connector: a small module that makes the encoder features readable by the LLM, for example an MLP, a Q-Former with learnable queries, or cross-attention inside the LLM
  • LLM: the language model backbone that does the reasoning
  • generator: an optional decoder, often a diffusion model, that creates images, audio or video from the LLM output

Why it is preferred

  • more than 90% of MLLMs belong to this category
  • it has a higher upper bound, because rich features reach the LLM directly instead of being reduced to text
  • all parts can be joined and trained as one unified model, so it is the more promising direction

Note

Architecture II is different from Architecture I in one main way. In Architecture I the LLM only talks to tools in text. In Architecture II the LLM directly consumes and produces multimodal representations.