Overview

  • benchmarks are the test sets used to judge whole MLLMs
  • six types, from classification and retrieval up to agentic tasks, and each type tests a different ability
  • warnings: one score can hide failures, and test data may have leaked into training (contamination)

MLLM benchmarks are the test sets used to evaluate multimodal models. Each type of benchmark probes a different ability, so a model must be checked on several types.

Benchmark types

TypeExamplesWhat it probes
Classification and retrievalImageNet zero-shot, COCO and Flickr retrievalencoder alignment quality
Visual question answeringVQAv2, GQA, TextVQA, MM-Vetgrounded reasoning, OCR, counting
Knowledge and reasoningMMMU, MathVistaexpert knowledge and multi-step reasoning
Compositionality and robustnessWinoground, ARO, ObjectNetword order, distribution shift
Human preferencepairwise arenas (for example 325K comparisons for Molmo)overall usefulness to people
Agentic tasksOSWorld, WebArena, AndroidWorldend-to-end task completion

Warnings

  • single numbers hide failure modes, so check the per-category results and the robustness sets
  • watch for contamination, which is benchmark data that the model already saw during pretraining
  • watch for scores that rise faster than real usefulness

Note

The agentic benchmarks test models that act, not only answer, see Multimodal Agents.