Overview

  • an encoder is the first block of an MLLM: it turns a raw image or sound into feature vectors
  • for images, CLIP-ViT is the most used, and SigLIP is a newer, smaller rival
  • main problem: ViTs only take small square images, so fine detail is lost, and two fixes are image slicing and dual-branch encoders
  • it ends with audio and 3D encoders, then unified encoders that put every modality in one space (ImageBind, LanguageBind)

A multimodal encoder is the first block of an MLLM. It takes a raw signal, such as an image or a sound clip, and turns it into a sequence of feature vectors that the rest of the model can use. Encoders are usually pretrained separately and then reused.

Visual encoder

CLIP-ViT is the most popular visual encoder for vision-language models.

  • it is a Vision Transformer that was trained with text, so its image features are already well aligned with the text space
  • it scales well when the number of parameters and the amount of data grow

SigLIP is gaining popularity because it is smaller and stronger than CLIP at the same job.

The plot below shows that bigger CLIP-style encoders (EVA-CLIP, DFN5B-CLIP, OpenCLIP, InternVL-C) reach a higher zero-shot accuracy.

Problem: fixed low resolution

Pretrained ViTs have one main limit:

  • they accept only a fixed low resolution in a square shape, usually 224 by 224 or 336 by 336 pixels
  • a large photo must be shrunk to fit, so fine-grained visual details are lost
  • high-resolution perception is essential, especially for OCR (reading text in an image)

High-resolution method 1: image slicing

Image slice-based models split a high-resolution image into smaller slices.

  1. the large image is split into a grid of slices, for example 2 by 2
  2. each slice is encoded separately at the encoder's normal resolution
  3. a resized copy of the whole image is also encoded, to keep the global view
  4. all the resulting tokens are flattened and passed to the LLM

Representatives: GPT-4V, LLaVA-NeXT, MiniCPM-V 2.0 and 2.5, LLaVA-UHD, mPLUG-DocOwl 1.5, SPHINX, InternLM-XComposer2-4KHD and Monkey.

High-resolution method 2: dual-branch encoders

Dual-branch models use two encoders at the same time.

  • a normal low-resolution encoder reads a downsampled image (for example 224 by 224) and feeds the main vision-language decoder
  • a lightweight high-resolution encoder reads the full image (for example 1120 by 1120)
  • the high-resolution features are injected into the decoder layers through cross-attention, so the model gets fine details without making the main token sequence much longer

Representatives: CogAgent, Mini-Gemini, DeepSeek-VL and LLaVA-HR.

Non-visual encoders

For audio:

  • Whisper: a Transformer encoder-decoder trained on speech; the audio is first converted to a log-Mel spectrogram, passed through two 1D convolution layers with GELU, and given sinusoidal position encodings
  • AudioCLIP: extends CLIP so that audio joins images and text in one space
  • HuBERT: a self-supervised speech encoder
  • BEATs: a self-supervised audio encoder for general sounds

For 3D point clouds:

  • Point-BERT: a Transformer encoder for point clouds

Unified multimodal encoders

Instead of one separate encoder per modality, a unified encoder puts all modalities into one shared space. When the different modalities are well aligned in that space, the LLM can understand all of them more easily.

ImageBind embeds all modalities into a joint space centred on images. It is described in ImageBind.

LanguageBind embeds all modalities into a joint space centred on language.

  • the language encoder is kept frozen, so language is the fixed anchor
  • each other modality (video, infrared, depth, audio, and more) has its own encoder that is trained to match the language features with contrastive learning
  • the other encoders are tuned with LoRA, and some of their input tokens are masked to save computation
  • a new modality can be added later by training one more encoder against the same language anchor, which is called modality extending