Overview
- these are standard tests of how good an encoder's features are
- five protocols: linear probe, zero-shot, retrieval (R@k), fine-tuning and dense tasks
- each one trains a different amount on top of the encoder, so each one measures something different
- key point: fine-tuning can hide a weak encoder, and zero-shot is the headline result for CLIP
A representation evaluation protocol is a standard test of how good an encoder's features are. Each protocol trains a different amount of extra model on top of the encoder, so each one measures a different property.
Five protocols
| Protocol | What you train | What it measures | Cost |
|---|---|---|---|
| Linear probe | a linear classifier on frozen features | how linearly separable the features are | low |
| Zero-shot | nothing | whether the space already matches language | none |
| Retrieval (R@k) | nothing | quality of the cross-modal alignment | none |
| Fine-tuning | the whole encoder | best achievable accuracy on one task | high |
| Dense tasks | a segmentation or detection head | spatial detail in the features | medium |
- a linear probe keeps the encoder frozen and trains only one linear layer, so a high score means the features already separate the classes
- R@k (recall at k) is the fraction of queries for which the correct match is among the top k retrieved items
Points to remember
- fine-tuning can hide a weak representation, because with enough data almost any encoder can be fixed
- CLIP's linear probe beats its zero-shot accuracy, yet zero-shot is the headline result because it needs no labels at all
- also report robustness (accuracy on shifted data) and efficiency (tokens, memory), not just one benchmark number
Note
These protocols judge the encoder alone. Whole MLLMs are judged with the benchmarks in MLLM Benchmarks.