Overview

  • here the LLM never sees the image or hears the audio, it only reads and writes text
  • it works like a manager: it plans the steps, then calls expert tools (captioner, detector, generator) with text commands
  • trade-off: cheap to build with a high lower bound, but no end-to-end training and a limited upper bound
  • main names: Visual-ChatGPT, HuggingGPT, MM-REACT, ViperGPT, AudioGPT and LLaVA-Plus

In Architecture I, the LLM acts as a discrete scheduler or controller. It only reads and writes text. It does not see the image or hear the audio itself. Instead, it decides which outside tool to call, and it tells that tool what to do in text.

How it works

  1. the user sends a request in text
  2. the LLM reads the request and plans the steps
  3. the LLM writes text commands that call downstream modules, which are separate expert models such as an image captioner, an object detector, an image generator or a speech recogniser
  4. each downstream module does its job on the actual image, audio or video
  5. the results return to the LLM as text, and the final answer goes back to the user

The word "discrete" means that the LLM and the modules only exchange discrete text, not continuous feature vectors.

Representative models

  • Visual-ChatGPT
  • HuggingGPT
  • MM-REACT
  • ViperGPT
  • AudioGPT
  • LLaVA-Plus

Strengths and weaknesses

  • no training is needed to connect the parts, so it is cheap and efficient to build
  • each module is already a strong expert, so the worst-case performance (the lower bound) is high
  • the system cannot be tuned end to end, because a gradient cannot pass through text messages
  • some multimodal information cannot be described well in text, so the best-case performance (the upper bound) is limited

Note

Because of the limited upper bound, most modern MLLMs use a different design in which the LLM receives multimodal features directly. That design is MLLM Architecture II (Encoder-LLM-Decoder).