AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts
Large Language Models (LLMs) based autonomous agents demonstrate multifaceted capabilities to contribute substantially to economic production. However, existing benchmarks remain focused on single agentic capability, failing to capture long-horizon real-world scenarios. Moreover, the reliance on human-in-the-loop feedback for realistic tasks creates a scalability bottleneck, hindering automated rollout collection and evaluation. To bridge this gap, we introduce AgencyBench, a comprehensive benchmark derived from daily AI usage, evaluating 6 core agentic capabilities across 32 real-world scenarios, comprising 138 tasks with specific queries, deliverables, and rubrics. These scenarios require an average of 90 tool calls, 1 million tokens, and hours of execution time to resolve. To enable automated evaluation, we employ a user simulation agent to provide iterative feedback, and a Docker sandbox to conduct visual and functional rubric-based assessment. Experiments reveal that closed-source models significantly outperform open-source models (48.4% vs 32.1%). Further analysis reveals significant disparities across models in resource efficiency, feedback-driven self-correction, and specific tool-use preferences. Finally, we investigate the impact of agentic scaffolds, observing that proprietary models demonstrate superior performance within their native ecosystems (e.g., Claude-4.5-Opus via Claude-Agent-SDK), while open-source models exhibit distinct performance peaks, suggesting potential optimization for specific execution frameworks. AgencyBench serves as a critical testbed for next-generation agents, highlighting the necessity of co-optimizing model architecture with agentic frameworks. We believe this work sheds light on the future direction of autonomous agents, and we release the full benchmark and evaluation toolkit at https://github.com/GAIR-NLP/AgencyBench.
696dd5da3f1837bfb8970a86
Keyu Li
claimed_verified
2026-01-19T09:22:02.662Z
668e476520e499a0786ea56e
https://cdn-avatars.huggingface.co/v1/production/uploads/668e476520e499a0786ea56e/lnvd1_UWW9o9ddrR6ehwR.png
Keyu Li (SII)
weizhihao1
user
weizhihao1
696dd5da3f1837bfb8970a87
Junhao Shi
696dd5da3f1837bfb8970a88
Yang Xiao
claimed_verified
2026-01-20T09:41:03.358Z
6002c316698168af3bb9f4a6
https://cdn-avatars.huggingface.co/v1/production/uploads/6002c316698168af3bb9f4a6/M2J2QFCRc5RhYRoz7OuZN.png
yangxiao
YangXiao-nlp
user
YangXiao-nlp
696dd5da3f1837bfb8970a89
Mohan Jiang
claimed_verified
2026-01-23T20:15:29.771Z
66d01e4401f2a6b4cd93ad87
https://cdn-avatars.huggingface.co/v1/production/uploads/66d01e4401f2a6b4cd93ad87/qxEUHyO8WauOCLcHXfiOS.png
Mohan Jiang (SII)
mhjiang0408
user
mhjiang0408
696dd5da3f1837bfb8970a8a
Jie Sun
claimed_verified
2026-01-20T09:40:59.146Z
64e6c617ecce34cb442cb208
/avatars/ebc61bf6a043314cb2089b1efd5e6a18.svg
JieSun(SII)
Sunshine279
user
Sunshine279
696dd5da3f1837bfb8970a8b
Yunze Wu
696dd5da3f1837bfb8970a8c
Shijie Xia
696dd5da3f1837bfb8970a8d
Xiaojie Cai
696dd5da3f1837bfb8970a8e
Tianze Xu
696dd5da3f1837bfb8970a8f
Weiye Si
696dd5da3f1837bfb8970a90
Wenjie Li
696dd5da3f1837bfb8970a91
Dequan Wang
claimed_verified
2026-01-20T09:41:01.349Z
61ad24836da53246bd6ac410
https://cdn-avatars.huggingface.co/v1/production/uploads/61ad24836da53246bd6ac410/o-FL-C6B77iB94wyAtTuO.png
Dequan Wang
dqwang
user
dqwang
696dd5da3f1837bfb8970a92
Pengfei Liu
AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts
Large Language Models (LLMs) based autonomous agents demonstrate multifaceted capabilities to contribute substantially to economic production. However, existing benchmarks remain focused on single agentic capability, failing to capture long-horizon real-world scenarios. Moreover, the reliance on human-in-the-loop feedback for realistic tasks creates a scalability bottleneck, hindering automated rollout collection and evaluation. To bridge this gap, we introduce AgencyBench, a comprehensive benchmark derived from daily AI usage, evaluating 6 core agentic capabilities across 32 real-world scenarios, comprising 138 tasks with specific queries, deliverables, and rubrics. These scenarios require an average of 90 tool calls, 1 million tokens, and hours of execution time to resolve. To enable automated evaluation, we employ a user simulation agent to provide iterative feedback, and a Docker sandbox to conduct visual and functional rubric-based assessment. Experiments reveal that closed-source models significantly outperform open-source models (48.4% vs 32.1%). Further analysis reveals significant disparities across models in resource efficiency, feedback-driven self-correction, and specific tool-use preferences. Finally, we investigate the impact of agentic scaffolds, observing that proprietary models demonstrate superior performance within their native ecosystems (e.g., Claude-4.5-Opus via Claude-Agent-SDK), while open-source models exhibit distinct performance peaks, suggesting potential optimization for specific execution frameworks. AgencyBench serves as a critical testbed for next-generation agents, highlighting the necessity of co-optimizing model architecture with agentic frameworks. We believe this work sheds light on the future direction of autonomous agents, and we release the full benchmark and evaluation toolkit at https://github.com/GAIR-NLP/AgencyBench.
AgencyBench presents a comprehensive benchmark for evaluating autonomous agents across real-world scenarios, enabling automated evaluation through user simulation and sandbox environments while revealing performance gaps between closed-source and open-source models.
large language models autonomous agents benchmarks agentic capabilities user simulation agent Docker sandbox tool calls token consumption model architecture agentic frameworks