Overview
- masked modelling hides part of the input and trains the model to fill it back in
- BERT hides about 15% of words, and MAE hides about 75% of image patches, because images repeat a lot
- MAE design: the big encoder sees only the visible patches, a small decoder rebuilds the rest, and the loss uses masked patches only
- it ends with masked modelling across modalities: MultiMAE and BEiT-3
Masked modelling hides part of the input and trains the model to reconstruct it. To fill the gaps, the model must understand structure and context, and no labels are needed. The Masked Autoencoder (MAE) applies this idea to images.
Hide and predict

- text (BERT): mask about 15% of the tokens, for example "The bat [MASK] noise at [MASK]"
- images (MAE): mask about 75% of the patches
Images need a much higher mask ratio because images are redundant: neighbouring pixels reveal the missing pixels, so a small mask would be too easy.
MAE architecture

- the image is cut into patches, and 75% of them are masked
- the encoder (a large ViT) sees only the visible 25% of the patches
- mask tokens are added back in the masked positions
- a small decoder rebuilds the pixels of every patch
- the loss is computed only on the masked patches
where:
- is the set of masked patches, and is how many there are
- is the true pixel values of patch
- is the reconstructed pixel values of patch
- is the squared error between them (mean squared error)
Key points:
- the encoder sees only 25% of the tokens, so pretraining is fast and memory-efficient
- the light decoder is discarded after pretraining, and only the encoder is kept
- the reconstructions look blurry, but the learned features are rich and fine-tune well
Masked modelling across modalities

MultiMAE:
- the inputs are RGB images, depth maps and semantic maps, each cut into patches
- a random subset of patches is kept, and all of them go into one shared Transformer
- there is one decoder per modality, which reconstructs its own patches
- predicting one modality from the others forces the model to use cross-modal information, which gives a joint representation
BEiT-3 ("image as a foreign language"): masked data modelling on images, texts and image-text pairs, with one shared multiway Transformer.
Note
Masked modelling needs no negatives, but it spends effort on rebuilding pixels. Self-Distillation predicts features instead of pixels.