Overview
- benchmarks are the test sets used to judge whole MLLMs
- six types, from classification and retrieval up to agentic tasks, and each type tests a different ability
- warnings: one score can hide failures, and test data may have leaked into training (contamination)
MLLM benchmarks are the test sets used to evaluate multimodal models. Each type of benchmark probes a different ability, so a model must be checked on several types.
Benchmark types
| Type | Examples | What it probes |
|---|---|---|
| Classification and retrieval | ImageNet zero-shot, COCO and Flickr retrieval | encoder alignment quality |
| Visual question answering | VQAv2, GQA, TextVQA, MM-Vet | grounded reasoning, OCR, counting |
| Knowledge and reasoning | MMMU, MathVista | expert knowledge and multi-step reasoning |
| Compositionality and robustness | Winoground, ARO, ObjectNet | word order, distribution shift |
| Human preference | pairwise arenas (for example 325K comparisons for Molmo) | overall usefulness to people |
| Agentic tasks | OSWorld, WebArena, AndroidWorld | end-to-end task completion |
Warnings
- single numbers hide failure modes, so check the per-category results and the robustness sets
- watch for contamination, which is benchmark data that the model already saw during pretraining
- watch for scores that rise faster than real usefulness
Note
The agentic benchmarks test models that act, not only answer, see Multimodal Agents.