Overview

  • these are standard tests of how good an encoder's features are
  • five protocols: linear probe, zero-shot, retrieval (R@k), fine-tuning and dense tasks
  • each one trains a different amount on top of the encoder, so each one measures something different
  • key point: fine-tuning can hide a weak encoder, and zero-shot is the headline result for CLIP

A representation evaluation protocol is a standard test of how good an encoder's features are. Each protocol trains a different amount of extra model on top of the encoder, so each one measures a different property.

Five protocols

ProtocolWhat you trainWhat it measuresCost
Linear probea linear classifier on frozen featureshow linearly separable the features arelow
Zero-shotnothingwhether the space already matches languagenone
Retrieval (R@k)nothingquality of the cross-modal alignmentnone
Fine-tuningthe whole encoderbest achievable accuracy on one taskhigh
Dense tasksa segmentation or detection headspatial detail in the featuresmedium
  • a linear probe keeps the encoder frozen and trains only one linear layer, so a high score means the features already separate the classes
  • R@k (recall at k) is the fraction of queries for which the correct match is among the top k retrieved items

Points to remember

  • fine-tuning can hide a weak representation, because with enough data almost any encoder can be fixed
  • CLIP's linear probe beats its zero-shot accuracy, yet zero-shot is the headline result because it needs no labels at all
  • also report robustness (accuracy on shifted data) and efficiency (tokens, memory), not just one benchmark number

Note

These protocols judge the encoder alone. Whole MLLMs are judged with the benchmarks in MLLM Benchmarks.