Overview
- multimodal learning is grouped by the problems it solves, not by which modalities it uses
- four core challenges: representation, alignment, fusion and translation
- key contrasts: joint against coordinated representation, and early, mid and late fusion
- read it first to get a frame for the method notes that follow
Multimodal learning is organised by the problems it must solve, not by the list of modalities. There are four core challenges, and every multimodal problem must solve at least one of them.
Four core challenges
- representation: what space or spaces are learned; either one joint space, or coordinated spaces with one space per modality
- alignment: how separate spaces are matched, so that related items end up close; the most common tool is Contrastive Learning
- fusion: how modalities are merged inside one model; tokens can be fused early, in the middle or late
- translation: how one modality is mapped to another; either retrieval (find an existing item) or generation (create a new one)
Joint against coordinated representations

Joint representation:
- the inputs of all modalities are combined, go through one model, and give one vector, written , where to are the inputs and is the shared model
- the modalities interact inside the model
- example: a video clip together with its audio track
Coordinated representation:
- there is one encoder per modality, and , and the training loss links the two outputs (shown as in the figure)
- similar items in different modalities end up close in the space
- example: CLIP, which has an image encoder and a text encoder
Fusion: where modalities meet

- early fusion: the tokens of all modalities are concatenated, then one shared Transformer processes them; rich interaction, but costly
- mid fusion: each modality first has separate layers, then cross-attention joins them; a balance of cost and interaction
- late fusion: separate models process each modality, and only their outputs are combined at the end; simple and modular, but there is little interaction
In Transformers, fusion is the layer where tokens start attending across modalities. Attention bottlenecks pass information between modalities through a few shared tokens only, which cuts the cost of early fusion.
Note
The methods used to solve the alignment challenge are compared in Multimodal Alignment Strategies.