Overview
- TACO teaches an MLLM to think, call a tool, read the result, and repeat
- this loop is called a chain of thought and action (CoTA)
- tools include OCR, LocalizeObjects, EstimateDepth, Crop, ZoomIn and Calculate
- key finding: fine-tuning on CoTA data works, but few-shot prompting with it does not
TACO trains an MLLM to solve hard visual questions with chains of thought and action (CoTA). The model does not answer at once. It thinks, calls a tool, reads the result, and repeats.
Problem
Open MLLMs still fail at:
- fine-grained OCR (reading small text in images)
- multi-step reasoning
- counting
- questions that need external knowledge
Chain of thought and action

- the model writes a thought, then calls a tool, then reads the result, and repeats until it calls Terminate(answer)
- the tools include OCR, LocalizeObjects, EstimateDepth, Crop, ZoomIn and Calculate
Example: "What is the price for tomatoes?" The correct answer is 8.0. Mantis-LLaVA answers 1.5 and LLaVA-OV answers 7.00, both wrong. TACO thinks "To determine the price of tomatoes, I need to extract the text", calls OCR on the image, reads that "DOMATES TINAZTEPE" has a price of "8,00", and calls Terminate("8.00").
Findings
- CoTA fine-tuning works, but few-shot prompting with CoTA examples does not
- it gives +1 to 4% on average over 8 benchmarks, and up to +15% on MM-Vet
- the quality of the synthetic data matters more than its quantity
Note
TACO is a data source for a multimodal agent.