AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarios
Real-world multimodal agents solve multi-step workflows grounded in visual evidence. For example, an agent can troubleshoot a device by linking a wiring photo to a schematic and validating the fix with online documentation, or plan a trip by interpreting a transit map and checking schedules under routing constraints. However, existing multimodal benchmarks mainly evaluate single-turn visual reasoning or specific tool skills, and they do not fully capture the realism, visual subtlety, and long-horizon tool use that practical agents require. We introduce AgentVista, a benchmark for generalist multimodal agents that spans 25 sub-domains across 7 categories, pairing realistic and detail-rich visual scenarios with natural hybrid tool use. Tasks require long-horizon tool interactions across modalities, including web search, image search, page navigation, and code-based operations for both image processing and general programming. Comprehensive evaluation of state-of-the-art models exposes significant gaps in their ability to carry out long-horizon multimodal tool use. Even the best model in our evaluation, Gemini-3-Pro with tools, achieves only 27.3% overall accuracy, and hard instances can require more than 25 tool-calling turns. We expect AgentVista to accelerate the development of more capable and reliable multimodal agents for realistic and ultra-challenging problem solving.
69aa5b47c4f6edddcc6c8721
Zhaochen Su
claimed_verified
2026-03-06T13:54:55.464Z
64264095ba51f8a2136946a0
https://cdn-avatars.huggingface.co/v1/production/uploads/64264095ba51f8a2136946a0/FR33boVpkDXcrvGMBmprF.jpeg
Zhaochen Su
Warrieryes
user
Warrieryes
69aa5b47c4f6edddcc6c8722
Jincheng Gao
69aa5b47c4f6edddcc6c8723
Hangyu Guo
69aa5b47c4f6edddcc6c8724
Zhenhua Liu
69aa5b47c4f6edddcc6c8725
Lueyang Zhang
69aa5b47c4f6edddcc6c8726
Xinyu Geng
69aa5b47c4f6edddcc6c8727
Shijue Huang
claimed_verified
2026-03-06T13:54:57.973Z
64ce05c631c655ff8a2e183c
/avatars/f2de7f8a1348b05f46946085e3e9718e.svg
Shijue Huang
JoeYing
user
JoeYing
69aa5b47c4f6edddcc6c8728
Peng Xia
claimed_verified
2026-03-19T13:04:07.874Z
643e9ee6f6bb3c31a26e7bc4
/avatars/acfaa7d6a23dada24c86b954c3be116a.svg
Peng Xia
richardxp888
user
richardxp888
69aa5b47c4f6edddcc6c8729
Guanyu Jiang
69aa5b47c4f6edddcc6c872a
Cheng Wang
69aa5b47c4f6edddcc6c872b
Yue Zhang
69aa5b47c4f6edddcc6c872c
Yi R. Fung
69aa5b47c4f6edddcc6c872d
Junxian He
AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarios
Real-world multimodal agents solve multi-step workflows grounded in visual evidence. For example, an agent can troubleshoot a device by linking a wiring photo to a schematic and validating the fix with online documentation, or plan a trip by interpreting a transit map and checking schedules under routing constraints. However, existing multimodal benchmarks mainly evaluate single-turn visual reasoning or specific tool skills, and they do not fully capture the realism, visual subtlety, and long-horizon tool use that practical agents require. We introduce AgentVista, a benchmark for generalist multimodal agents that spans 25 sub-domains across 7 categories, pairing realistic and detail-rich visual scenarios with natural hybrid tool use. Tasks require long-horizon tool interactions across modalities, including web search, image search, page navigation, and code-based operations for both image processing and general programming. Comprehensive evaluation of state-of-the-art models exposes significant gaps in their ability to carry out long-horizon multimodal tool use. Even the best model in our evaluation, Gemini-3-Pro with tools, achieves only 27.3% overall accuracy, and hard instances can require more than 25 tool-calling turns. We expect AgentVista to accelerate the development of more capable and reliable multimodal agents for realistic and ultra-challenging problem solving.
AgentVista presents a comprehensive benchmark for multimodal agents requiring long-horizon tool interactions across multiple modalities and complex real-world scenarios.
multimodal agents visual reasoning tool use benchmark long-horizon tasks multimodal tool interactions hybrid tool use visual scenarios