Overview

  • this note is about turning each kind of signal into vectors, so that vectors of the same thing land close together
  • it sets the scene: encoder, connector and LLM, and what an embedding is
  • three ways to learn representations: supervised, self-supervised and weakly supervised
  • then a tour by modality: text (word2vec to BERT and GPT), vision (CNN to ViT, then a self-supervised timeline), audio, and one common recipe table

Multimodal representation learning is the study of how to turn each kind of signal (text, images, audio and more) into vectors, and how to relate the vectors of different signals to each other. It is the base of every MLLM, because the LLM can only work with vectors.

Where it fits in an MLLM

Most MLLMs follow an Encoder, Connector, LLM design, where the LLM is the "brain":

  • each modality (image, audio, video) has its own encoder that turns the signal into vectors
  • the connector (projector) makes these vectors readable by the LLM
  • the LLM reads them together with the text and writes a text response

This leaves two questions: how to learn good vectors for each modality, and how to align them.

Why representation and alignment matter

A toddler learns "cat" from several channels at once: hearing the word, seeing the animal and hearing "meow". All three signals point to one concept.

Deep networks only accept numbers, so every signal must first become a vector. This gives two core problems:

  • representation: how to encode each input as numbers
  • combination: how to relate or merge the different modalities

The goal is that different signals of the same thing end up as nearby vectors.

What an embedding is

An embedding is a vector produced by a neural network.

  • the distance between two vectors can be measured, and a small distance means a similar meaning
  • a good embedding space groups similar things together, is useful for many downstream tasks, and is learned from data with backpropagation
  • in a multimodal space, the word "cat" and a photo of a cat should land close together

In this 2D picture, text embeddings (circles) and image embeddings (squares) of the same concept sit close together, and animals form one group while vehicles form another. Real embeddings have hundreds of dimensions.

Three ways to learn representations

  • supervised: human labels define the task, for example ImageNet classification; labels are costly and fixed to one task
  • self-supervised: the data creates its own task by hiding, predicting or comparing parts of itself, for example BERT, SimCLR, MAE and DINO; no labels are needed, so it scales to huge unlabelled data
  • weakly (naturally) supervised: naturally paired data from the web is used, for example CLIP with 400M image-text pairs; it is cheap at scale, but the pairs are noisy

Modern foundation models mostly use self-supervision and weak supervision.

Language 1: static word embeddings

  • word2vec learns vectors by predicting words from their neighbours
  • CBOW: the context words predict the centre word
  • Skip-gram: the centre word predicts the context words
  • GloVe builds vectors from global word co-occurrence counts
  • similar words get similar vectors
  • limitation: there is only one vector per word, so "bank" (river) and "bank" (money) get the same vector

Language 2: contextual embeddings

A Transformer lets each token attend to all other tokens, so a word's vector depends on its context.

In "I sat by the river bank", the vector for bank is close to "shore". In "I got a bank loan today", it is close to "finance". Same word, different vectors: this is the key improvement over word2vec.

Language 3: two pretraining objectives

  • BERT, masked language modelling: about 15% of tokens are hidden and predicted, for example "The [MASK] made noise" predicts "bat"; the model sees both directions, so it is strong for understanding tasks
  • GPT, next-token prediction: the model predicts the next token, for example "The bat made" predicts "noise"; it sees the left context only, and it is the basis of today's LLMs

Vision 1: CNNs and ResNet

  • convolutions detect local patterns, and deeper layers combine them, from edges to textures to parts to whole objects
  • ResNet adds skip connections, , where is the block input and is what the layers in the block compute; this makes very deep networks trainable
  • ImageNet-pretrained CNNs were the default image encoder for years

Vision 2: Vision Transformer

A Vision Transformer (ViT) treats an image as a sequence of "visual words" and uses the same Transformer as text. This unifies vision and language processing, so ViT is the backbone of CLIP, MAE, DINO and most MLLMs.

Vision 3: self-supervised learning timeline

  • 2020, SimCLR: contrastive learning that compares two views of an image
  • 2020, BYOL: learning without negative pairs
  • 2021, DINO: self-distillation with a ViT
  • 2022, MAE: mask 75% of the patches and rebuild them
  • 2023, DINOv2: trained on 142M curated images
  • 2023, I-JEPA: predict in feature space instead of pixel space
  • 2025, DINOv3: 1.7B images; self-supervised learning now beats weak supervision
  • 2025, V-JEPA 2: trained on more than 1M hours of video

The trend is from comparing views, to masking, to predicting features at huge scale.

Audio

Audio is first turned into a spectrogram, then encoded much like an image, see Audio Representations.

Common recipe across modalities

ModalityRaw signalTokensTypical encoderSelf-supervised task
Textcharacterssub-word tokensTransformermasked or next-token prediction
Imagepixels (H x W x 3)16 x 16 patchesViT (or CNN)contrastive, masked patches, self-distillation
Audiowaveformspectrogram patches or framesCNN plus Transformercontrastive, masked prediction
Videoframes over timespace-time patchesvideo ViTmasked prediction in feature space

Everything becomes a sequence of tokens, so one Transformer design can process any modality.

Beyond text, images and audio

  • text and images are only snapshots of our continuous world
  • humans also sense motion, depth, touch, temperature and balance
  • sensors turn these into digital signals that a model can learn from, such as IMU (motion), 3D, depth, touch and video
  • making one model per sensor does not scale, so the field is organised by challenges, not by modalities, see Multimodal Learning Challenges