Overview

  • the connector (or projector) is a small bridge that turns encoder features into vectors the LLM can read like words
  • five connector types sit in three families: one token per patch (linear, MLP), learned queries (Q-Former, Perceiver Resampler) and convolution (C-Abstractor)
  • key point: the number of visual tokens matters, and only the last two families reduce it
  • learn which model uses which connector, because these pairs are easy to mix up

Input-side projection is the step that connects a modality encoder to the LLM. The encoder gives feature vectors in its own space. The LLM only understands vectors in its own word-embedding space. A small module, called the connector or projector, maps the encoder features into the LLM semantic space so that the LLM can read them like word tokens.

Five connector types at a glance

The figure shows three common shapes: an MLP (left), a Q-Former with learnable queries (middle), and cross-attention inside the LLM (right).

  • linear projection: used by LLaVA, MiniGPT-4 and NExT-GPT
  • two-layer MLP: used by LLaVA-1.5 and LLaVA-NeXT, CogVLM, DeepSeek-VL and Yi-VL
  • Q-Former: used by BLIP-2, InstructBLIP, VisCPM and VisualGLM
  • Perceiver Resampler: used by Flamingo, Qwen-VL, MiniCPM-V and LLaVA-UHD
  • C-Abstractor: used by HoneyBee and MM1

All five do one job: take the patch vectors from the encoder, for example 576 vectors with 1024 channels each, and turn them into tokens in the LLM's hidden dimension. They differ in how many tokens come out, which splits them into three families.

Family 1: one token per patch (linear and MLP)

The flattening of each patch's pixels is already done inside the encoder, so the connector receives one ready-made vector per patch.

Linear projection

  • one weight matrix:
  • the shape of sets the output dimension: with of shape , a 1024-channel patch vector becomes a 4096-channel token
  • each of the 4096 outputs is a weighted sum of all 1024 inputs, so "linear" describes how each output is made, not how many outputs there are
  • going up to 4096 adds no new information, it only puts the features into the LLM's coordinates
  • there is no activation function, so it can only stretch, rotate and mix the input directions

Two-layer MLP

  • linear layer, then a non-linearity, then a second linear layer:
  • it has one hidden layer, for example in LLaVA-1.5
  • the non-linearity is the point: two linear layers with nothing between them collapse into one linear layer, since is just another matrix
  • so the MLP can learn a curved mapping, which translates encoder features into the LLM space more flexibly

What both have in common:

  • each patch vector is mapped on its own, so the token count does not change: 576 patches in gives 576 tokens out
  • in practice, the same is applied to all 576 rows in one matrix multiply

Family 2: learned queries (Q-Former and Perceiver Resampler)

  • a fixed set of learned query vectors reads all the patch vectors through cross-attention
  • each query outputs one vector, so you get as many tokens as there are queries

Why the output count follows the queries, with 32 queries and 576 patch vectors:

  • the queries come from the 32 learned query vectors, so 32 rows
  • the keys and values come from the 576 patch vectors, so 576 rows
  • the attention scores have shape , and each row says how much one query cares about each patch
  • a softmax across each row turns the scores into weights that add up to 1
  • the output is the weights times : gives
  • so each query outputs one weighted mix of all 576 patches, and the patches only supply the information to read from

In self-attention over the patches, the queries come from the 576 patches themselves, so 576 vectors come out. That is why a ViT and a linear or MLP connector keep 576 tokens.

How the two differ:

  • Q-Former (BLIP-2): 32 queries, and its first training stage also uses text, so the queries learn to pick out image parts that match a caption
  • Perceiver Resampler (Flamingo): 64 queries, vision only, and built to take many video frames at once, with the features of every frame joined into one long list first

Weak point: a query can mix patches from anywhere in the image, so the spatial layout (which patch sits next to which) gets blurred.

Family 3: convolution (C-Abstractor)

  • it does not use queries, and it keeps the patch vectors in their spatial layout, for example
  • convolution layers mix each patch with its neighbours
  • pooling then shrinks the layout, for example tokens
  • strength: each output token still stands for one local region, so the spatial layout is kept, which is what "keeping local spatial context" means

Feeding the LLM through cross-attention

Most connectors put their tokens into the LLM's input sequence, next to the text tokens. Flamingo does not.

  • its Perceiver Resampler output is not put into the input sequence
  • new gated cross-attention layers placed between the blocks of the frozen LLM attend to the visual tokens
  • the gates start closed (at zero), so the frozen LLM is not damaged at the start of training
  • other models that use a resampler, such as Qwen-VL, MiniCPM-V and LLaVA-UHD, put its output tokens into the input sequence like normal tokens

A worked example of the simplest connector, and of how it is trained in two stages, is in LLaVA.

Findings about which connector is best

Different papers reach different conclusions:

  • a two-layer MLP is better than a linear projection (LLaVA-1.5); in its ablation, changing the linear layer to an MLP connector raised the MME score from about 1324 to about 1355 and the GQA score from 46.8 to 47.3
  • the resampler is comparable to the C-Abstractor (MM1) and to the MLP (LLaVA-UHD)

All papers agree on one point:

  • the number of visual tokens matters, especially for efficiency
  • the Resampler, Q-Former and C-Abstractor give fewer visual tokens than an MLP or a linear layer, because they compress the features into a fixed small set
  • fewer tokens per image is an advantage for high-resolution image understanding, where an image may otherwise need thousands of tokens

Why the token count matters

Every visual token takes a place in the LLM context, like a word. An MLP keeps one token per image patch, so a detailed image can fill the context and make the LLM slow. A query-based connector always outputs the same small number of tokens, which keeps the cost fixed.

Note

The input side is only half of the connection. The way the LLM sends information to output decoders is described in Decoding-side Connection.