The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic,
  and Long-Horizon Task Execution
Real-world language agents must handle complex, multi-step workflows across
diverse Apps. For instance, an agent may manage emails by coordinating with
calendars and file systems, or monitor a production database to detect
anomalies and generate reports following an operating manual. However, existing
language agent benchmarks often focus on narrow domains or simplified tasks
that lack the diversity, realism, and long-horizon complexity required to
evaluate agents' real-world performance. To address this gap, we introduce the
Tool Decathlon (dubbed as Toolathlon), a benchmark for language agents offering
diverse Apps and tools, realistic environment setup, and reliable
execution-based evaluation. Toolathlon spans 32 software applications and 604
tools, ranging from everyday platforms such as Google Calendar and Notion to
professional ones like WooCommerce, Kubernetes, and BigQuery. Most of the tools
are based on a high-quality set of Model Context Protocol (MCP) servers that we
may have revised or implemented ourselves. Unlike prior works, which primarily
ensure functional realism but offer limited environment state diversity, we
provide realistic initial environment states from real software, such as Canvas
courses with dozens of students or real financial spreadsheets. This benchmark
includes 108 manually sourced or crafted tasks in total, requiring interacting
with multiple Apps over around 20 turns on average to complete. Each task is
strictly verifiable through dedicated evaluation scripts. Comprehensive
evaluation of SOTA models highlights their significant shortcomings: the
best-performing model, Claude-4.5-Sonnet, achieves only a 38.6% success rate
with 20.2 tool calling turns on average, while the top open-weights model
DeepSeek-V3.2-Exp reaches 20.1%. We expect Toolathlon to drive the development
of more capable language agents for real-world, long-horizon task execution.
6902bdbe72739622ee92a654
Junlong Li
6902bdbe72739622ee92a655
Wenshuo Zhao
6902bdbe72739622ee92a656
Jian Zhao
6902bdbe72739622ee92a657
Weihao Zeng
claimed_verified
2025-10-31T14:28:07.277Z
62751082b43ccfeef483424f
/avatars/fec83e4478e7d1731ba6033328131852.svg
WeihaoZeng
AndrewZeng
user
AndrewZeng
6902bdbe72739622ee92a658
Haoze Wu
claimed_verified
2025-10-30T14:32:48.349Z
657c2a82429a20edb51d2029
https://cdn-avatars.huggingface.co/v1/production/uploads/657c2a82429a20edb51d2029/OkQVj4GAnWzMAkif-mpod.jpeg
Haoze Wu
WaitHZ
user
WaitHZ
6902bdbe72739622ee92a659
Xiaochen Wang
6902bdbe72739622ee92a65a
Rui Ge
6902bdbe72739622ee92a65b
Yuxuan Cao
6902bdbe72739622ee92a65c
Yuzhen Huang
claimed_verified
2025-10-30T14:32:38.988Z
6462def82a83863b97c0611e
/avatars/c03e9cc7d75b0266fcc56ecb6ee62148.svg
Yuzhen Huang
yuzhen17
user
yuzhen17
6902bdbe72739622ee92a65d
Wei Liu
claimed_verified
2025-10-30T14:32:45.233Z
6458af46f4d212d780bd7c68
/avatars/832fd34bcc041b0b7b551873a459fc3c.svg
Wei Liu
PeterV09
user
PeterV09
6902bdbe72739622ee92a65e
Junteng Liu
6902bdbe72739622ee92a65f
Zhaochen Su
6902bdbe72739622ee92a660
Yiyang Guo
6902bdbe72739622ee92a661
Fan Zhou
6902bdbe72739622ee92a662
Lueyang Zhang
6902bdbe72739622ee92a663
Juan Michelini
6902bdbe72739622ee92a664
Xingyao Wang
6902bdbe72739622ee92a665
Xiang Yue
claimed_verified
2025-12-16T22:33:18.870Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
yuexiang96
user
yuexiang96
6902bdbe72739622ee92a666
Shuyan Zhou
6902bdbe72739622ee92a667
Graham Neubig
6902bdbe72739622ee92a668
Junxian He
The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic,
  and Long-Horizon Task Execution
Real-world language agents must handle complex, multi-step workflows across
diverse Apps. For instance, an agent may manage emails by coordinating with
calendars and file systems, or monitor a production database to detect
anomalies and generate reports following an operating manual. However, existing
language agent benchmarks often focus on narrow domains or simplified tasks
that lack the diversity, realism, and long-horizon complexity required to
evaluate agents' real-world performance. To address this gap, we introduce the
Tool Decathlon (dubbed as Toolathlon), a benchmark for language agents offering
diverse Apps and tools, realistic environment setup, and reliable
execution-based evaluation. Toolathlon spans 32 software applications and 604
tools, ranging from everyday platforms such as Google Calendar and Notion to
professional ones like WooCommerce, Kubernetes, and BigQuery. Most of the tools
are based on a high-quality set of Model Context Protocol (MCP) servers that we
may have revised or implemented ourselves. Unlike prior works, which primarily
ensure functional realism but offer limited environment state diversity, we
provide realistic initial environment states from real software, such as Canvas
courses with dozens of students or real financial spreadsheets. This benchmark
includes 108 manually sourced or crafted tasks in total, requiring interacting
with multiple Apps over around 20 turns on average to complete. Each task is
strictly verifiable through dedicated evaluation scripts. Comprehensive
evaluation of SOTA models highlights their significant shortcomings: the
best-performing model, Claude-4.5-Sonnet, achieves only a 38.6% success rate
with 20.2 tool calling turns on average, while the top open-weights model
DeepSeek-V3.2-Exp reaches 20.1%. We expect Toolathlon to drive the development
of more capable language agents for real-world, long-horizon task execution.
Toolathlon is a benchmark for language agents that evaluates their performance across diverse, realistic, and complex multi-step workflows involving multiple software applications and tools.
Tool Decathlon Toolathlon Model Context Protocol MCP Google Calendar Notion WooCommerce Kubernetes BigQuery Canvas financial spreadsheets evaluation scripts Claude-4.5-Sonnet DeepSeek-V3.2-Exp