Overview

  • this note is a map of four training families that build and match representations
  • the four: contrastive learning, masked modelling, self-distillation and LLM-centric alignment, each with examples
  • a table says which family to pick for which data and need
  • it ends with recent trends such as SigLIP 2, DINOv3 and V-JEPA 2

Alignment strategies are the training methods used to make good representations and to match modalities with each other. There are four main families. Modern encoders often combine several of them, for example SigLIP 2.

Big picture

  • contrastive learning: pull matched pairs together and push other pairs apart; examples are SimCLR, CLIP, SigLIP and ImageBind, see Contrastive Learning
  • masked modelling: hide part of the input and reconstruct it; examples are BERT, MAE, MultiMAE and BEiT-3, see Masked Autoencoder (MAE)
  • self-distillation (latent prediction): a student network predicts a teacher network's features; examples are BYOL, DINO, data2vec and JEPA, see Self-Distillation
  • LLM-centric alignment: project the features into the LLM's token space; examples are LLaVA, BLIP-2, Flamingo and Qwen-VL, see Input-side Projection

Which strategy to use

If you have or needConsider
Paired data, and you need retrieval or zero-shot useContrastive learning (CLIP, SigLIP)
Unlabelled single-modality data, and you need fine detailMasked modelling (MAE) or self-distillation (DINO)
Several synchronised sensors, and you need one fused vectorJoint masked modelling or early fusion
Few pairs for a new modalityBind it to an existing hub (ImageBind style)
Language reasoning over signalsAligned encoder, plus a connector, plus two-stage tuning
  • deeper vision-language fusion: Qwen3-VL (DeepStack) injects ViT features from several levels into several early LLM layers, instead of only at the input
  • native resolution and long context: ViTs that accept dynamic resolutions; Qwen3-VL handles 256K tokens of interleaved text, image and video
  • encoders with mixed objectives: SigLIP 2 combines sigmoid contrastive learning, captioning, self-distillation and masked prediction
  • self-supervision catches up: DINOv3 matches or beats weakly supervised encoders on many vision tasks
  • from perception to action: V-JEPA 2 turns video features into a world model for robot planning