When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for
  Visual Chain-of-Thought
We propose MIRA, a new benchmark designed to evaluate models in scenarios
where generating intermediate visual images is essential for successful
reasoning. Unlike traditional CoT methods that rely solely on text, tasks in
MIRA require models to generate and utilize intermediate images - such as
sketches, structural diagrams, or path drawings - to guide their reasoning
process. This setup closely mirrors how humans solve complex problems through
"drawing to think". To solve this, MIRA focuses on tasks that are intrinsically
challenging and involve complex structures, spatial relationships, or reasoning
steps that are difficult to express through language alone. To ensure that our
evaluation data is of high-quality, we include 546 multimodal problems,
annotated with intermediate visual images and final answers. We also propose a
unified evaluation protocol for MIRA that spans three levels of evaluation
input: direct input with image and question only, text-only CoT input with
image and thinking prompts, and Visual-CoT input with both annotated image
clues and textual thinking prompts. To probe the upper bound of model capacity
on our benchmark, we also report pass@k and majority voting accuracies under
different k settings. Experimental results show that existing multimodal large
language models, including strongest private models as well as strong
open-weight models, perform poorly when relying solely on textual prompts.
However, when intermediate visual cues are provided, model performance improves
consistently, yielding an average relative gain of 33.7% across all models and
tasks. We also probe the upper bound by expanding the search space and
designing textual prompts aligned with Visual-CoT, but both yield only limited
improvements compared to our Visual-CoT setting. These results underscore the
critical role of imagined visual information in enabling successful reasoning
on MIRA.
690adcf8d70e173c84529049
Yiyang Zhou
690adcf8d70e173c8452904a
Haoqin Tu
690adcf8d70e173c8452904b
Zijun Wang
claimed_verified
2026-04-07T08:50:20.910Z
64efe4cf82c6eea604baffb2
https://cdn-avatars.huggingface.co/v1/production/uploads/64efe4cf82c6eea604baffb2/SLmCIJqLlEdOPNvTCgFR4.jpeg
Zijun Wang
Olivia714
user
Olivia714
690adcf8d70e173c8452904c
Zeyu Wang
690adcf8d70e173c8452904d
Niklas Muennighoff
690adcf8d70e173c8452904e
Fan Nie
claimed_verified
2026-01-27T09:44:41.715Z
652fb9e437fad005c70d66f4
https://cdn-avatars.huggingface.co/v1/production/uploads/652fb9e437fad005c70d66f4/xjvro1bFvQ8ILR0tkD6qU.jpeg
Fan Nie
FanNie1208
user
FanNie1208
690adcf8d70e173c8452904f
Yejin Choi
690adcf8d70e173c84529050
James Zou
690adcf8d70e173c84529051
Chaorui Deng
690adcf8d70e173c84529052
Shen Yan
690adcf8d70e173c84529053
Haoqi Fan
690adcf8d70e173c84529054
Cihang Xie
690adcf8d70e173c84529055
Huaxiu Yao
690adcf8d70e173c84529056
Qinghao Ye
When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for
  Visual Chain-of-Thought
We propose MIRA, a new benchmark designed to evaluate models in scenarios
where generating intermediate visual images is essential for successful
reasoning. Unlike traditional CoT methods that rely solely on text, tasks in
MIRA require models to generate and utilize intermediate images - such as
sketches, structural diagrams, or path drawings - to guide their reasoning
process. This setup closely mirrors how humans solve complex problems through
"drawing to think". To solve this, MIRA focuses on tasks that are intrinsically
challenging and involve complex structures, spatial relationships, or reasoning
steps that are difficult to express through language alone. To ensure that our
evaluation data is of high-quality, we include 546 multimodal problems,
annotated with intermediate visual images and final answers. We also propose a
unified evaluation protocol for MIRA that spans three levels of evaluation
input: direct input with image and question only, text-only CoT input with
image and thinking prompts, and Visual-CoT input with both annotated image
clues and textual thinking prompts. To probe the upper bound of model capacity
on our benchmark, we also report pass@k and majority voting accuracies under
different k settings. Experimental results show that existing multimodal large
language models, including strongest private models as well as strong
open-weight models, perform poorly when relying solely on textual prompts.
However, when intermediate visual cues are provided, model performance improves
consistently, yielding an average relative gain of 33.7% across all models and
tasks. We also probe the upper bound by expanding the search space and
designing textual prompts aligned with Visual-CoT, but both yield only limited
improvements compared to our Visual-CoT setting. These results underscore the
critical role of imagined visual information in enabling successful reasoning
on MIRA.
MIRA is a benchmark that evaluates models using intermediate visual images to enhance reasoning, showing significant performance improvements over text-only methods.
CoT methods intermediate visual images sketches structural diagrams path drawings multimodal problems evaluation protocol pass@k majority voting accuracies multimodal large language models Visual-CoT input