Overview
- here the LLM takes in the image or audio features directly, so it is one part of a joined system
- shape: encoder, then LLM, then decoder, with a small connector between the encoder and the LLM
- this is the main design today (over 90% of MLLMs), because it has a higher upper bound and can be trained as one model
- read it side by side with Architecture I, since the two are often compared
In Architecture II, the LLM is a joint part of the system. It receives multimodal information directly from outside, understands it by itself, and then passes instructions to decoders or generators that produce non-text outputs. The structure is Encoder, LLM, Decoder.
Three parts

- encoder side: text goes straight into the LLM, while each other signal (image, audio, video) first goes through a multimodal encoder that turns it into feature vectors
- core: the LLM reads the text and the encoded features together, and reasons about them
- decoder side: text comes straight out of the LLM, while a multimodal decoder turns the LLM output into an image, audio or video when one is needed
Components in detail

A typical model of this type has four trainable or frozen blocks:
- modality encoder: a pretrained network, such as a vision encoder, that turns raw input into feature vectors, see Multimodal Encoders
- connector: a small module that makes the encoder features readable by the LLM, for example an MLP, a Q-Former with learnable queries, or cross-attention inside the LLM
- LLM: the language model backbone that does the reasoning
- generator: an optional decoder, often a diffusion model, that creates images, audio or video from the LLM output
Why it is preferred
- more than 90% of MLLMs belong to this category
- it has a higher upper bound, because rich features reach the LLM directly instead of being reduced to text
- all parts can be joined and trained as one unified model, so it is the more promising direction
Note
Architecture II is different from Architecture I in one main way. In Architecture I the LLM only talks to tools in text. In Architecture II the LLM directly consumes and produces multimodal representations.