Exploring Reasoning Reward Model for Agents
Agentic Reinforcement Learning (Agentic RL) has achieved notable success in enabling agents to perform complex reasoning and tool use. However, most methods still relies on sparse outcome-based reward for training. Such feedback fails to differentiate intermediate reasoning quality, leading to suboptimal training results. In this paper, we introduce Agent Reasoning Reward Model (Agent-RRM), a multi-faceted reward model that produces structured feedback for agentic trajectories, including (1) an explicit reasoning trace , (2) a focused critique that provides refinement guidance by highlighting reasoning flaws, and (3) an overall score that evaluates process performance. Leveraging these signals, we systematically investigate three integration strategies: Reagent-C (text-augmented refinement), Reagent-R (reward-augmented guidance), and Reagent-U (unified feedback integration). Extensive evaluations across 12 diverse benchmarks demonstrate that Reagent-U yields substantial performance leaps, achieving 43.7% on GAIA and 46.2% on WebWalkerQA, validating the effectiveness of our reasoning reward model and training schemes. Code, models, and datasets are all released to facilitate future research.
697c2a85a67238fac88cc129
Kaixuan Fan
claimed_verified
2026-01-30T09:36:18.527Z
6656b9a6ca7f02c3d4817744
/avatars/01d32967cad6054389c61b5bb9a528b1.svg
kxbunny
bunny127
user
bunny127
697c2a85a67238fac88cc12a
Kaituo Feng
697c2a85a67238fac88cc12b
Manyuan Zhang
697c2a85a67238fac88cc12c
Tianshuo Peng
697c2a85a67238fac88cc12d
Zhixun Li
697c2a85a67238fac88cc12e
Yilei Jiang
697c2a85a67238fac88cc12f
Shuang Chen
697c2a85a67238fac88cc130
Peng Pei
697c2a85a67238fac88cc131
Xunliang Cai
697c2a85a67238fac88cc132
Xiangyu Yue
Exploring Reasoning Reward Model for Agents
Agentic Reinforcement Learning (Agentic RL) has achieved notable success in enabling agents to perform complex reasoning and tool use. However, most methods still relies on sparse outcome-based reward for training. Such feedback fails to differentiate intermediate reasoning quality, leading to suboptimal training results. In this paper, we introduce Agent Reasoning Reward Model (Agent-RRM), a multi-faceted reward model that produces structured feedback for agentic trajectories, including (1) an explicit reasoning trace , (2) a focused critique that provides refinement guidance by highlighting reasoning flaws, and (3) an overall score that evaluates process performance. Leveraging these signals, we systematically investigate three integration strategies: Reagent-C (text-augmented refinement), Reagent-R (reward-augmented guidance), and Reagent-U (unified feedback integration). Extensive evaluations across 12 diverse benchmarks demonstrate that Reagent-U yields substantial performance leaps, achieving 43.7% on GAIA and 46.2% on WebWalkerQA, validating the effectiveness of our reasoning reward model and training schemes. Code, models, and datasets are all released to facilitate future research.
Agent-RRM, a multi-faceted reward model, provides structured feedback for agentic trajectories through reasoning traces, critiques, and performance scores, with unified feedback integration showing superior performance across diverse benchmarks.
Agentic Reinforcement Learning reward model reasoning trace critique overall score Reagent-C Reagent-R Reagent-U GAIA WebWalkerQA