Benchmark Test-Time Scaling of General LLM Agents
LLM agents are increasingly expected to function as general-purpose systems capable of resolving open-ended user requests. While existing benchmarks focus on domain-aware environments for developing specialized agents, evaluating general-purpose agents requires more realistic settings that challenge them to operate across multiple skills and tools within a unified environment. We introduce General AgentBench, a benchmark that provides such a unified framework for evaluating general LLM agents across search, coding, reasoning, and tool-use domains. Using General AgentBench, we systematically study test-time scaling behaviors under sequential scaling (iterative interaction) and parallel scaling (sampling multiple trajectories). Evaluation of ten leading LLM agents reveals a substantial performance degradation when moving from domain-specific evaluations to this general-agent setting. Moreover, we find that neither scaling methodology yields effective performance improvements in practice, due to two fundamental limitations: context ceiling in sequential scaling and verification gap in parallel scaling. Code is publicly available at https://github.com/cxcscmu/General-AgentBench.
699f4416ebfce7fbcca91a1b
Xiaochuan Li
699f4416ebfce7fbcca91a1c
Ryan Ming
699f4416ebfce7fbcca91a1d
Pranav Setlur
699f4416ebfce7fbcca91a1e
Abhijay Paladugu
699f4416ebfce7fbcca91a1f
Andy Tang
699f4416ebfce7fbcca91a20
Hao Kang
699f4416ebfce7fbcca91a21
Shuai Shao
699f4416ebfce7fbcca91a22
Rong Jin
699f4416ebfce7fbcca91a23
Chenyan Xiong
Benchmark Test-Time Scaling of General LLM Agents
LLM agents are increasingly expected to function as general-purpose systems capable of resolving open-ended user requests. While existing benchmarks focus on domain-aware environments for developing specialized agents, evaluating general-purpose agents requires more realistic settings that challenge them to operate across multiple skills and tools within a unified environment. We introduce General AgentBench, a benchmark that provides such a unified framework for evaluating general LLM agents across search, coding, reasoning, and tool-use domains. Using General AgentBench, we systematically study test-time scaling behaviors under sequential scaling (iterative interaction) and parallel scaling (sampling multiple trajectories). Evaluation of ten leading LLM agents reveals a substantial performance degradation when moving from domain-specific evaluations to this general-agent setting. Moreover, we find that neither scaling methodology yields effective performance improvements in practice, due to two fundamental limitations: context ceiling in sequential scaling and verification gap in parallel scaling. Code is publicly available at https://github.com/cxcscmu/General-AgentBench.
General AgentBench evaluates large language model agents across multiple domains and scaling methods, revealing performance degradation and fundamental limitations in sequential and parallel scaling approaches.
LLM agents general-purpose systems domain-aware environments unified environment test-time scaling sequential scaling parallel scaling context ceiling verification gap