AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents
While Large Language Models (LLMs) have evolved into tool-using agents, they remain brittle in long-horizon interactions. Unlike mathematical reasoning where errors are often rectifiable via backtracking, tool-use failures frequently induce irreversible side effects, making accurate step-level verification critical. However, existing process-level benchmarks are predominantly confined to closed-world mathematical domains, failing to capture the dynamic and open-ended nature of tool execution. To bridge this gap, we introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories. The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement. It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity. Extensive experiments reveal key insights: (1) weaker policy models exhibit inflated ratios of correct steps due to early termination; (2) distinguishing neutral and erroneous actions remains a significant challenge for current models; and (3) process-derived signals provide complementary value to outcome supervision, significantly enhancing test-time scaling. We hope AgentProcessBench can foster future research in reward models and pave the way toward general agents. The code and data are available at https://github.com/RUCBM/AgentProcessBench.
69ba01095d2b9c28ea09ef9c
Shengda Fan
69ba01095d2b9c28ea09ef9d
Xuyan Ye
claimed_verified
2026-03-18T08:45:28.239Z
68905a353cf91a8e828fd8a1
https://cdn-avatars.huggingface.co/v1/production/uploads/68905a353cf91a8e828fd8a1/lyXMELqDGOHNlXeYlTL5X.jpeg
Xuyan Ye
LulaCola
user
LulaCola
69ba01095d2b9c28ea09ef9e
Yupeng Huo
admin_assigned
2026-03-18T10:03:27.724Z
65bc66b77c6ad1994b7eb00e
/avatars/4559dd2b44695eeb48ed66051cfea98a.svg
yupeng huo
YupengHuo
user
YupengHuo
69ba01095d2b9c28ea09ef9f
Zhi-Yuan Chen
admin_assigned
2026-03-18T10:03:33.179Z
65309a1d657ae56cdb65e0e7
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/lHQI9RNjfz8E5v1uyCGeV.png
Zhi-Yuan Chen
JaxChen
user
JaxChen
69ba01095d2b9c28ea09efa0
Yiju Guo
admin_assigned
2026-03-18T10:03:39.573Z
64bb937d8496ee0fb6cac9aa
https://cdn-avatars.huggingface.co/v1/production/uploads/64bb937d8496ee0fb6cac9aa/oFkKNxaMrd3wAciwP4Lu5.png
YijuGuo
YijuGuo
user
YijuGuo
69ba01095d2b9c28ea09efa1
Shenzhi Yang
69ba01095d2b9c28ea09efa2
Wenkai Yang
admin_assigned
2026-03-18T10:03:51.311Z
64b7df742f5a966b973e25f7
/avatars/e24e7769188d441317b3b7d10ef8fd60.svg
Wenkai Yang
Keven16
user
Keven16
69ba01095d2b9c28ea09efa3
Shuqi Ye
admin_assigned
2026-03-18T10:03:57.398Z
698ab2ebc9804eab58756f66
/avatars/797aa01a039a42671b8140c7742c71a5.svg
ShuqiYe
ShuqiYe
user
ShuqiYe
69ba01095d2b9c28ea09efa4
Jingwen Chen
69ba01095d2b9c28ea09efa5
Haotian Chen
69ba01095d2b9c28ea09efa6
Xin Cong
69ba01095d2b9c28ea09efa7
Yankai Lin
admin_assigned
2026-03-18T10:04:10.938Z
657a651e1433ea7d44de6397
/avatars/ccfc76f94595a38ff4a80f77c911eabf.svg
Yankai Lin
lyk423
user
lyk423
AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents
While Large Language Models (LLMs) have evolved into tool-using agents, they remain brittle in long-horizon interactions. Unlike mathematical reasoning where errors are often rectifiable via backtracking, tool-use failures frequently induce irreversible side effects, making accurate step-level verification critical. However, existing process-level benchmarks are predominantly confined to closed-world mathematical domains, failing to capture the dynamic and open-ended nature of tool execution. To bridge this gap, we introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories. The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement. It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity. Extensive experiments reveal key insights: (1) weaker policy models exhibit inflated ratios of correct steps due to early termination; (2) distinguishing neutral and erroneous actions remains a significant challenge for current models; and (3) process-derived signals provide complementary value to outcome supervision, significantly enhancing test-time scaling. We hope AgentProcessBench can foster future research in reward models and pave the way toward general agents. The code and data are available at https://github.com/RUCBM/AgentProcessBench.
AgentProcessBench introduces a benchmark for evaluating step-level effectiveness in tool-augmented agent interactions, featuring diverse trajectories with detailed human annotations to improve process-level understanding and model performance.
Large Language Models tool-using agents long-horizon interactions mathematical reasoning backtracking tool execution AgentProcessBench step-level verification process-level benchmarks closed-world domains open-ended nature diverse trajectories human-labeled step annotations inter-annotator agreement ternary labeling scheme error propagation rule policy models early termination reward models test-time scaling