TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images
  Reasoning
The frontier of visual reasoning is shifting toward models like OpenAI o3,
which can intelligently create and operate tools to transform images for
problem-solving, also known as thinking-with-images in
chain-of-thought. Yet existing benchmarks fail to fully capture this advanced
capability. Even Visual Search, the most common benchmark for current
thinking-with-images methods, tests only basic operations such as
localization and cropping, offering little insight into more complex, dynamic,
and tool-dependent reasoning. We introduce TIR-Bench, a comprehensive
benchmark for evaluating agentic thinking-with-images across 13 diverse tasks,
each requiring novel tool use for image processing and manipulation in
chain-of-thought. We evaluate 22 multimodal large language models (MLLMs), from
leading open-sourced and proprietary models to those with explicit tool-use
augmentation. Results show that TIR-Bench is universally challenging, and
strong performance requires genuine thinking-with-images capabilities. Finally,
we present a pilot study comparing direct versus agentic fine-tuning.
6909788f812eca10f9cc6257
Ming Li
6909788f812eca10f9cc6258
Jike Zhong
6909788f812eca10f9cc6259
Shitian Zhao
claimed_verified
2025-11-04T20:25:00.119Z
62c66504031996c36c86976a
https://cdn-avatars.huggingface.co/v1/production/uploads/62c66504031996c36c86976a/wIq0YJhkWnEhlzsh-TGYO.png
steve z
stzhao
user
stzhao
6909788f812eca10f9cc625a
Haoquan Zhang
claimed_verified
2025-11-04T20:25:02.529Z
67ff7f687351095d4b606b84
https://cdn-avatars.huggingface.co/v1/production/uploads/67ff7f687351095d4b606b84/KhNPmbBC3zghuP5h1MK-c.png
Haoquan Zhang
haoquan03
user
haoquan03
6909788f812eca10f9cc625b
Shaoheng Lin
6909788f812eca10f9cc625c
Yuxiang Lai
6909788f812eca10f9cc625d
Wei Chen
6909788f812eca10f9cc625e
Konstantinos Psounis
6909788f812eca10f9cc625f
Kaipeng Zhang
claimed_verified
2026-03-25T10:52:34.020Z
65f1713552c38a91e0a445e8
/avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
kaipeng
kpzhang996
user
kpzhang996
TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images
  Reasoning
The frontier of visual reasoning is shifting toward models like OpenAI o3,
which can intelligently create and operate tools to transform images for
problem-solving, also known as thinking-with-images in
chain-of-thought. Yet existing benchmarks fail to fully capture this advanced
capability. Even Visual Search, the most common benchmark for current
thinking-with-images methods, tests only basic operations such as
localization and cropping, offering little insight into more complex, dynamic,
and tool-dependent reasoning. We introduce TIR-Bench, a comprehensive
benchmark for evaluating agentic thinking-with-images across 13 diverse tasks,
each requiring novel tool use for image processing and manipulation in
chain-of-thought. We evaluate 22 multimodal large language models (MLLMs), from
leading open-sourced and proprietary models to those with explicit tool-use
augmentation. Results show that TIR-Bench is universally challenging, and
strong performance requires genuine thinking-with-images capabilities. Finally,
we present a pilot study comparing direct versus agentic fine-tuning.
TIR-Bench evaluates advanced visual reasoning capabilities in multimodal models through diverse tasks requiring tool use and chain-of-thought, demonstrating the need for genuine thinking-with-images.
OpenAI o3 thinking-with-images chain-of-thought TIR-Bench multimodal large language models MLLMs tool-use augmentation direct fine-tuning agentic fine-tuning