AdaTooler-V: Adaptive Tool-Use for Images and Videos
Recent advances have shown that multimodal large language models (MLLMs) benefit from multimodal interleaved chain-of-thought (CoT) with vision tool interactions. However, existing open-source models often exhibit blind tool-use reasoning patterns, invoking vision tools even when they are unnecessary, which significantly increases inference overhead and degrades model performance. To this end, we propose AdaTooler-V, an MLLM that performs adaptive tool-use by determining whether a visual problem truly requires tools. First, we introduce AT-GRPO, a reinforcement learning algorithm that adaptively adjusts reward scales based on the Tool Benefit Score of each sample, encouraging the model to invoke tools only when they provide genuine improvements. Moreover, we construct two datasets to support training: AdaTooler-V-CoT-100k for SFT cold start and AdaTooler-V-300k for RL with verifiable rewards across single-image, multi-image, and video data. Experiments across twelve benchmarks demonstrate the strong reasoning capability of AdaTooler-V, outperforming existing methods in diverse visual reasoning tasks. Notably, AdaTooler-V-7B achieves an accuracy of 89.8\% on the high-resolution benchmark V*, surpassing the commercial proprietary model GPT-4o and Gemini 1.5 Pro. All code, models, and data are released.
6944d2c9fbf17e708e1860e3
Chaoyang Wang
6944d2c9fbf17e708e1860e4
Kaituo Feng
6944d2c9fbf17e708e1860e5
Dongyang Chen
6944d2c9fbf17e708e1860e6
Zhongyu Wang
6944d2c9fbf17e708e1860e7
Zhixun Li
6944d2c9fbf17e708e1860e8
Sicheng Gao
6944d2c9fbf17e708e1860e9
Meng Meng
6944d2c9fbf17e708e1860ea
Xu Zhou
6944d2c9fbf17e708e1860eb
Manyuan Zhang
6944d2c9fbf17e708e1860ec
Yuzhang Shang
6944d2c9fbf17e708e1860ed
Xiangyu Yue
AdaTooler-V: Adaptive Tool-Use for Images and Videos
Recent advances have shown that multimodal large language models (MLLMs) benefit from multimodal interleaved chain-of-thought (CoT) with vision tool interactions. However, existing open-source models often exhibit blind tool-use reasoning patterns, invoking vision tools even when they are unnecessary, which significantly increases inference overhead and degrades model performance. To this end, we propose AdaTooler-V, an MLLM that performs adaptive tool-use by determining whether a visual problem truly requires tools. First, we introduce AT-GRPO, a reinforcement learning algorithm that adaptively adjusts reward scales based on the Tool Benefit Score of each sample, encouraging the model to invoke tools only when they provide genuine improvements. Moreover, we construct two datasets to support training: AdaTooler-V-CoT-100k for SFT cold start and AdaTooler-V-300k for RL with verifiable rewards across single-image, multi-image, and video data. Experiments across twelve benchmarks demonstrate the strong reasoning capability of AdaTooler-V, outperforming existing methods in diverse visual reasoning tasks. Notably, AdaTooler-V-7B achieves an accuracy of 89.8\% on the high-resolution benchmark V*, surpassing the commercial proprietary model GPT-4o and Gemini 1.5 Pro. All code, models, and data are released.
AdaTooler-V, a multimodal large language model, adaptively uses vision tools based on reinforcement learning, improving performance and reducing unnecessary tool invocations in visual reasoning tasks.
multimodal large language models multimodal interleaved chain-of-thought vision tool interactions reinforcement learning adaptive tool-use Tool Benefit Score AdaTooler-V-CoT-100k AdaTooler-V-300k visual reasoning tasks high-resolution benchmark