M3-Bench: Multi-Modal, Multi-Hop, Multi-Threaded Tool-Using MLLM Agent Benchmark
We present M^3-Bench, the first benchmark for evaluating multimodal tool use under the Model Context Protocol. The benchmark targets realistic, multi-hop and multi-threaded workflows that require visual grounding and textual reasoning, cross-tool dependencies, and persistence of intermediate resources across steps. We introduce a similarity-driven alignment that serializes each tool call, embeds signatures with a sentence encoder, and performs similarity-bucketed Hungarian matching to obtain auditable one-to-one correspondences. On top of this alignment, we report interpretable metrics that decouple semantic fidelity from workflow consistency. The benchmark spans 28 servers with 231 tools, and provides standardized trajectories curated through an Executor & Judge pipeline with human verification; an auxiliary four large language models (LLMs) judge ensemble reports end-task Task Completion and information grounding. Evaluations of representative state-of-the-art Multimodal LLMs (MLLMs) reveal persistent gaps in multimodal MCP tool use, particularly in argument fidelity and structure consistency, underscoring the need for methods that jointly reason over images, text, and tool graphs. Our Benchmark's anonymous repository is at https://github.com/EtaYang10th/Open-M3-Bench
6925154e16eb3a9f1310390b
Yang Zhou
6925154e16eb3a9f1310390c
Mingyu Zhao
6925154e16eb3a9f1310390d
Zhenting Wang
6925154e16eb3a9f1310390e
Difei Gu
claimed_verified
2025-11-25T09:05:43.437Z
64fd1fa384bf01577e90382c
/avatars/43b39461781b504c08831f2cdc3af432.svg
Difei Gu
difeigu
user
difeigu
6925154e16eb3a9f1310390f
Bangwei Guo
6925154e16eb3a9f13103910
Ruosong Ye
6925154e16eb3a9f13103911
Ligong Han
6925154e16eb3a9f13103912
Can Jin
6925154e16eb3a9f13103913
Dimitris N. Metaxas
M3-Bench: Multi-Modal, Multi-Hop, Multi-Threaded Tool-Using MLLM Agent Benchmark
We present M^3-Bench, the first benchmark for evaluating multimodal tool use under the Model Context Protocol. The benchmark targets realistic, multi-hop and multi-threaded workflows that require visual grounding and textual reasoning, cross-tool dependencies, and persistence of intermediate resources across steps. We introduce a similarity-driven alignment that serializes each tool call, embeds signatures with a sentence encoder, and performs similarity-bucketed Hungarian matching to obtain auditable one-to-one correspondences. On top of this alignment, we report interpretable metrics that decouple semantic fidelity from workflow consistency. The benchmark spans 28 servers with 231 tools, and provides standardized trajectories curated through an Executor & Judge pipeline with human verification; an auxiliary four large language models (LLMs) judge ensemble reports end-task Task Completion and information grounding. Evaluations of representative state-of-the-art Multimodal LLMs (MLLMs) reveal persistent gaps in multimodal MCP tool use, particularly in argument fidelity and structure consistency, underscoring the need for methods that jointly reason over images, text, and tool graphs. Our Benchmark's anonymous repository is at https://github.com/EtaYang10th/Open-M3-Bench
M^3-Bench evaluates multimodal tool use with a focus on visual grounding, textual reasoning, and tool dependencies using a novel similarity-driven alignment method and interpretable metrics.
Model Context Protocol multimodal tool use multi-hop workflows multi-threaded workflows visual grounding textual reasoning cross-tool dependencies intermediate resources similarity-driven alignment sentence encoder Hungarian matching semantic fidelity workflow consistency Multimodal LLMs argument fidelity structure consistency tool graphs