ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents
Reward-guided search methods have demonstrated strong potential in enhancing tool-using agents by effectively guiding sampling and exploration over complex action spaces. As a core design, those search methods utilize process reward models (PRMs) to provide step-level rewards, enabling more fine-grained monitoring. However, there is a lack of systematic and reliable evaluation benchmarks for PRMs in tool-using settings. In this paper, we introduce ToolPRMBench, a large-scale benchmark specifically designed to evaluate PRMs for tool-using agents. ToolPRMBench is built on top of several representative tool-using benchmarks and converts agent trajectories into step-level test cases. Each case contains the interaction history, a correct action, a plausible but incorrect alternative, and relevant tool metadata. We respectively utilize offline sampling to isolate local single-step errors and online sampling to capture realistic multi-step failures from full agent rollouts. A multi-LLM verification pipeline is proposed to reduce label noise and ensure data quality. We conduct extensive experiments across large language models, general PRMs, and tool-specialized PRMs on ToolPRMBench. The results reveal clear differences in PRM effectiveness and highlight the potential of specialized PRMs for tool-using. Code and data will be released at https://github.com/David-Li0406/ToolPRMBench.
697055b4a8be625b19c2af18
Dawei Li
admin_assigned
2026-01-21T12:00:07.519Z
6474e1afb68461d5cf7c41cc
https://cdn-avatars.huggingface.co/v1/production/uploads/6474e1afb68461d5cf7c41cc/bcoiD_qPrjHUBlB259djg.png
Dawei Li
wjldw
user
wjldw
697055b4a8be625b19c2af19
Yuguang Yao
admin_assigned
2026-01-21T12:00:12.781Z
644cb05d778ecbfb9783fd8b
https://cdn-avatars.huggingface.co/v1/production/uploads/644cb05d778ecbfb9783fd8b/N_F53QPr4u8ilxGFC_f0V.jpeg
Yuguang Yao
yaoyugua
user
yaoyugua
697055b4a8be625b19c2af1a
Zhen Tan
697055b4a8be625b19c2af1b
Huan Liu
697055b4a8be625b19c2af1c
Ruocheng Guo
admin_assigned
2026-01-21T12:00:23.178Z
65dcb410bda21d181b38321b
/avatars/a9caed79c4eb14352b4015377fcae1d7.svg
Ruocheng Guo
rguo12
user
rguo12
ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents
Reward-guided search methods have demonstrated strong potential in enhancing tool-using agents by effectively guiding sampling and exploration over complex action spaces. As a core design, those search methods utilize process reward models (PRMs) to provide step-level rewards, enabling more fine-grained monitoring. However, there is a lack of systematic and reliable evaluation benchmarks for PRMs in tool-using settings. In this paper, we introduce ToolPRMBench, a large-scale benchmark specifically designed to evaluate PRMs for tool-using agents. ToolPRMBench is built on top of several representative tool-using benchmarks and converts agent trajectories into step-level test cases. Each case contains the interaction history, a correct action, a plausible but incorrect alternative, and relevant tool metadata. We respectively utilize offline sampling to isolate local single-step errors and online sampling to capture realistic multi-step failures from full agent rollouts. A multi-LLM verification pipeline is proposed to reduce label noise and ensure data quality. We conduct extensive experiments across large language models, general PRMs, and tool-specialized PRMs on ToolPRMBench. The results reveal clear differences in PRM effectiveness and highlight the potential of specialized PRMs for tool-using. Code and data will be released at https://github.com/David-Li0406/ToolPRMBench.
ToolPRMBench is introduced as a large-scale benchmark for evaluating process reward models in tool-using agents, featuring step-level test cases and multi-LLM verification to ensure data quality.
process reward models tool-using agents reward-guided search agent trajectories step-level rewards large language models multi-LLM verification offline sampling online sampling