Overview

  • an MLLM is a language model that can also take in images, sound, video and 3D data, and sometimes make them
  • big idea: the world has many kinds of signals, and language models already work very well, so keep the LLM as the "brain" and add the other senses around it
  • this note walks through how the brain joins senses, why language models came first (scaling law), real uses that need many signals, the trend of models over time, six key aspects, and why language sits at the core
  • read it as the map for the other multimodal LLM notes

A Multimodal Large Language Model (MLLM) is a large language model that can also take in, and sometimes produce, signals other than text, such as images, audio, video and 3D data. The goal is to give a language model the same deep understanding of these signals that it already has of language.

Starting point: how a human brain senses

A human does not learn about the world from text alone. The brain receives many senses at the same time and joins them together.

  • sensing: smell, hearing, vision, touch, a sense of space, a sense of time, and attention that selects what to process
  • function: perception and cognition, emotion (happy or sad), imagining "what if", taking action, and learning from reward and from the environment
  • memory: short-term memory and long-term memory, with the hippocampus acting like a hash table that stores and finds memories
  • architectural features: sparse activation (only a few neurons fire at a time), event-driven processing, predictive coding, and distributed and parallel computation

Each sense organ uses neurons to convert an analog stimulus (light, sound, pressure, a chemical) into a spike train, which is a series of electrical pulses. So the brain receives every sense as the same kind of signal. This idea of "turn every signal into one common form" is also the idea behind an MLLM.

Why language models came first

Language models became powerful first, for two reasons.

  • scaling law: when the model size and the data size both increase, the test loss decreases in a predictable way
  • data size bottleneck: if the data stays small, a bigger model soon stops getting better, so the data must grow together with the model
  • sample efficiency: a larger model needs fewer training tokens to reach the same loss as a smaller model

Language models then evolved very fast. The number of research papers that mention "language model" or "large language model" grew slowly from 2018 to 2021, then grew very sharply after ChatGPT appeared in late 2022. Important models along the way include BERT, GPT-1, GPT-2, T5, GPT-3, Codex, InstructGPT, ChatGPT, LLaMA and GPT-4.

World is multimodal

The world contains much more than language. Useful signals include:

  • language, documents and tables, code
  • images, video, depth and 3D data
  • sound and spectrograms
  • time series, graphs and heat maps
  • infrared and radar, touch and smell

Real systems already need several of these at the same time:

  • autonomous driving systems use cameras (visual), LiDAR (spatial data) and sonar (audio signals) to drive safely
  • healthcare diagnostics use MRI, CT and X-ray images, together with patient history and spoken symptoms, to find diseases
  • smart home assistants such as Amazon Alexa and Google Home use voice commands (audio), physical interaction (touch) and visual cues

From LLM to MLLM

The central question is: can the success of LLMs be transferred to MLLMs, so that the model understands multimodal information as deeply as it understands language?

  • an LLM takes text in and gives text out
  • an MLLM takes text, image, audio, video and more in, and can give one or more of these out

The belief behind this work is that a model which perceives and interacts with the world as humans do may be the key to human-level AI.

The number of MLLMs grew very fast from 2022 to 2024.

  • early models in 2022 to early 2023 include Flamingo, BLIP-2, Kosmos-1, PaLM-E, LLaVA, MiniGPT-4 and mPLUG-Owl
  • systems that call other tools, such as Visual ChatGPT, HuggingGPT, MM-REACT and ViperGPT, also appeared in early 2023
  • by 2024 there were many models for video (Video-LLaVA, LLaMA-VID), 3D (3D-LLM, PointLLM), grounding (Kosmos-2, Shikra) and any-to-any generation (NExT-GPT, AnyGPT)
  • from late 2023 a branch of efficient MLLMs appeared, which are small enough for phones and edge devices: TinyGPT-V, Gemini Nano, MobileVLM, Vary-toy, LLaVA-Phi, MoE-LLaVA, Imp, ALLaVA, TinyLLaVA, SPHINX-Tiny, MobileVLMv2, Bunny, DeepSeek-VL, MiniCPM-V, VL-Mamba, MM1, LLaVA-Gemma, Mipha, Cobra, Mini-Gemini and moondream

Key aspects for building a powerful MLLM

Six aspects work together towards human-level AI.

  • architecture: what is the current design of an MLLM, what modalities does it support, and how can MLLMs be grouped
  • instruction tuning: how the model learns to follow instructions that involve other modalities
  • modality and functionality: what tasks MLLMs can do, and what the current progress of the field is
  • hallucination: why an MLLM says things that are not in the image or audio, which kinds of hallucination are common, and how to reduce them
  • evaluation: which methods and datasets are used to test MLLMs, and what the weak points of these benchmarks are
  • human-level reasoning: reasoning about multimodal input as well as a human does

Language as the core

Language models already show emergent phenomena, which are abilities that appear only when the model is large enough. They also have very strong semantic understanding. This suggests that language is a crucial modality for carrying intelligence.

For this reason, every current MLLM uses a language-based LLM as its core decision-making module, which is its "brain" or central processor. Other modalities (visual, video, 3D, sound, touch, smell) are added around this core by external modules that are not text-based. These added modules give the LLM its multimodal abilities.

There are two main ways to connect these modules to the LLM. The first way keeps the LLM as a text-only controller, described in MLLM Architecture I (LLM as Scheduler). The second way makes the LLM part of one joint system.