OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents
With advances in decision-making and reasoning capabilities, multimodal
agents show strong potential in computer application scenarios. Past
evaluations have mainly assessed GUI interaction skills, while tool invocation
abilities, such as those enabled by the Model Context Protocol (MCP), have been
largely overlooked. Comparing agents with integrated tool invocation to those
evaluated only on GUI interaction is inherently unfair. We present OSWorld-MCP,
the first comprehensive and fair benchmark for assessing computer-use agents'
tool invocation, GUI operation, and decision-making abilities in a real-world
environment. We design a novel automated code-generation pipeline to create
tools and combine them with a curated selection from existing tools. Rigorous
manual validation yields 158 high-quality tools (covering 7 common
applications), each verified for correct functionality, practical
applicability, and versatility. Extensive evaluations of state-of-the-art
multimodal agents on OSWorld-MCP show that MCP tools generally improve task
success rates (e.g., from 8.3% to 20.4% for OpenAI o3 at 15 steps, from 40.1%
to 43.3% for Claude 4 Sonnet at 50 steps), underscoring the importance of
assessing tool invocation capabilities. However, even the strongest models have
relatively low tool invocation rates, Only 36.3%, indicating room for
improvement and highlighting the benchmark's challenge. By explicitly measuring
MCP tool usage skills, OSWorld-MCP deepens understanding of multimodal agents
and sets a new standard for evaluating performance in complex, tool-assisted
environments. Our code, environment, and data are publicly available at
https://osworld-mcp.github.io.
690172eb646208eac0d1f357
Hongrui Jia
claimed_verified
2026-02-27T13:58:36.177Z
66a4c04555677524a0c8047b
/avatars/0913eadc4e33eab9ac705e1ad4df9fc9.svg
Hongrui Jia
hongruijia
user
hongruijia
690172eb646208eac0d1f358
Jitong Liao
690172eb646208eac0d1f359
Xi Zhang
690172eb646208eac0d1f35a
Haiyang Xu
claimed_verified
2025-10-29T12:45:53.210Z
645b10e80c73ea27d13f7aca
/avatars/95e565306472a15067440b5b43e07a6f.svg
xuhaiyang
xhyandwyy
user
xhyandwyy
690172eb646208eac0d1f35b
Tianbao Xie
690172eb646208eac0d1f35c
Chaoya Jiang
690172eb646208eac0d1f35d
Ming Yan
690172eb646208eac0d1f35e
Si Liu
690172eb646208eac0d1f35f
Wei Ye
690172eb646208eac0d1f360
Fei Huang
OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents
With advances in decision-making and reasoning capabilities, multimodal
agents show strong potential in computer application scenarios. Past
evaluations have mainly assessed GUI interaction skills, while tool invocation
abilities, such as those enabled by the Model Context Protocol (MCP), have been
largely overlooked. Comparing agents with integrated tool invocation to those
evaluated only on GUI interaction is inherently unfair. We present OSWorld-MCP,
the first comprehensive and fair benchmark for assessing computer-use agents'
tool invocation, GUI operation, and decision-making abilities in a real-world
environment. We design a novel automated code-generation pipeline to create
tools and combine them with a curated selection from existing tools. Rigorous
manual validation yields 158 high-quality tools (covering 7 common
applications), each verified for correct functionality, practical
applicability, and versatility. Extensive evaluations of state-of-the-art
multimodal agents on OSWorld-MCP show that MCP tools generally improve task
success rates (e.g., from 8.3% to 20.4% for OpenAI o3 at 15 steps, from 40.1%
to 43.3% for Claude 4 Sonnet at 50 steps), underscoring the importance of
assessing tool invocation capabilities. However, even the strongest models have
relatively low tool invocation rates, Only 36.3%, indicating room for
improvement and highlighting the benchmark's challenge. By explicitly measuring
MCP tool usage skills, OSWorld-MCP deepens understanding of multimodal agents
and sets a new standard for evaluating performance in complex, tool-assisted
environments. Our code, environment, and data are publicly available at
https://osworld-mcp.github.io.
OSWorld-MCP is a benchmark that evaluates multimodal agents' tool invocation, GUI operation, and decision-making abilities, highlighting the importance of assessing tool usage in real-world scenarios.
multimodal agents Model Context Protocol (MCP) OSWorld-MCP automated code-generation pipeline GUI operation task success rates tool invocation rates