VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chaining
Recent advancements extend Multimodal Large Language Models (MLLMs) beyond standard visual question answering to utilizing external tools for advanced visual tasks. Despite this progress, precisely executing and effectively composing diverse tools for complex tasks remain persistent bottleneck. Constrained by sparse tool-sets and simple tool-use trajectories, existing benchmarks fail to capture complex and diverse tool interactions, falling short in evaluating model performance under practical, real-world conditions. To bridge this gap, we introduce VisualToolChain-Bench(VTC-Bench), a comprehensive benchmark designed to evaluate tool-use proficiency in MLLMs. To align with realistic computer vision pipelines, our framework features 32 diverse OpenCV-based visual operations. This rich tool-set enables extensive combinations, allowing VTC-Bench to rigorously assess multi-tool composition and long-horizon, multi-step plan execution. For precise evaluation, we provide 680 curated problems structured across a nine-category cognitive hierarchy, each with ground-truth execution trajectories. Extensive experiments on 19 leading MLLMs reveal critical limitations in current models' visual agentic capabilities. Specifically, models struggle to adapt to diverse tool-sets and generalize to unseen operations, with the leading model Gemini-3.0-Pro only achieving 51% on our benchmark. Furthermore, multi-tool composition remains a persistent challenge. When facing complex tasks, models struggle to formulate efficient execution plans, relying heavily on a narrow, suboptimal subset of familiar functions rather than selecting the optimal tools. By identifying these fundamental challenges, VTC-Bench establishes a rigorous baseline to guide the development of more generalized visual agentic models.
69bcb965081b9ad49af57299
Xuanyu Zhu
claimed_verified
2026-03-23T10:53:23.309Z
644d2532d185572dd1e48f90
/avatars/5831acebb02d8bc8f80f56b7b11c7c69.svg
Zhu
zzzhu
user
zzzhu
69bcb965081b9ad49af5729a
Yuhao Dong
claimed_verified
2026-03-23T10:53:03.713Z
652965773a416e1f2173443b
https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg
Yuhao Dong
THUdyh
user
THUdyh
69bcb965081b9ad49af5729b
Rundong Wang
69bcb965081b9ad49af5729c
Yang Shi
claimed_verified
2026-03-23T10:53:21.229Z
673c7319d11b1c2e246ead9c
https://cdn-avatars.huggingface.co/v1/production/uploads/673c7319d11b1c2e246ead9c/IjFIO--N7Hm_BOEafhEQv.jpeg
Yang Shi
DogNeverSleep
user
DogNeverSleep
69bcb965081b9ad49af5729d
Zhipeng Wu
69bcb965081b9ad49af5729e
Yinlun Peng
69bcb965081b9ad49af5729f
YiFan Zhang
69bcb965081b9ad49af572a0
Yihang Lou
69bcb965081b9ad49af572a1
Yuanxing Zhang
69bcb965081b9ad49af572a2
Ziwei Liu
69bcb965081b9ad49af572a3
Yan Bai
69bcb965081b9ad49af572a4
Yuan Zhou
VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chaining
Recent advancements extend Multimodal Large Language Models (MLLMs) beyond standard visual question answering to utilizing external tools for advanced visual tasks. Despite this progress, precisely executing and effectively composing diverse tools for complex tasks remain persistent bottleneck. Constrained by sparse tool-sets and simple tool-use trajectories, existing benchmarks fail to capture complex and diverse tool interactions, falling short in evaluating model performance under practical, real-world conditions. To bridge this gap, we introduce VisualToolChain-Bench(VTC-Bench), a comprehensive benchmark designed to evaluate tool-use proficiency in MLLMs. To align with realistic computer vision pipelines, our framework features 32 diverse OpenCV-based visual operations. This rich tool-set enables extensive combinations, allowing VTC-Bench to rigorously assess multi-tool composition and long-horizon, multi-step plan execution. For precise evaluation, we provide 680 curated problems structured across a nine-category cognitive hierarchy, each with ground-truth execution trajectories. Extensive experiments on 19 leading MLLMs reveal critical limitations in current models' visual agentic capabilities. Specifically, models struggle to adapt to diverse tool-sets and generalize to unseen operations, with the leading model Gemini-3.0-Pro only achieving 51% on our benchmark. Furthermore, multi-tool composition remains a persistent challenge. When facing complex tasks, models struggle to formulate efficient execution plans, relying heavily on a narrow, suboptimal subset of familiar functions rather than selecting the optimal tools. By identifying these fundamental challenges, VTC-Bench establishes a rigorous baseline to guide the development of more generalized visual agentic models.
A new benchmark called VisualToolChain-Bench is introduced to evaluate the tool-use capabilities of multimodal large language models in complex visual tasks requiring multi-step planning and diverse tool composition.
Multimodal Large Language Models tool-use proficiency visual question answering OpenCV-based visual operations multi-tool composition long-horizon execution cognitive hierarchy ground-truth execution trajectories visual agentic capabilities