MM-InstructEval is the multimodal version of the same idea. It treats the instruction itself as an experimental variable, and uses ten instruction templates per task.
Scale
- 45 models were evaluated, of which 36 are multimodal LLMs
Four metrics
- Best Performance, which is peak capability
- Mean Relative Gain
- Stability, which measures robustness to instruction wording
- Adaptability, which scores model and instruction compatibility
Findings
- the same model can swing widely as the instruction wording changes, and Stability is what quantifies that
- open MLLMs break on multi-step tasks such as PuzzleVQA and long-tail relation extraction, and not on basic captioning
- MultiInstruct combines 62 tasks from 21 open source datasets
Reading a result
- a model that tops a benchmark and then falls when instructions are reworded has high Best Performance and low Stability, so the prompt is part of the system
The text only version is InstructEval.