Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration
Search-integrated reasoning enables language agents to transcend static parametric knowledge by actively querying external sources. However, training these agents via reinforcement learning is hindered by the multi-scale credit assignment problem: existing methods typically rely on sparse, trajectory-level rewards that fail to distinguish between high-quality reasoning and fortuitous guesses, leading to redundant or misleading search behaviors. To address this, we propose Search-R2, a novel Actor-Refiner collaboration framework that enhances reasoning through targeted intervention, with both components jointly optimized during training. Our approach decomposes the generation process into an Actor, which produces initial reasoning trajectories, and a Meta-Refiner, which selectively diagnoses and repairs flawed steps via a 'cut-and-regenerate' mechanism. To provide fine-grained supervision, we introduce a hybrid reward design that couples outcome correctness with a dense process reward quantifying the information density of retrieved evidence. Theoretically, we formalize the Actor-Refiner interaction as a smoothed mixture policy, proving that selective correction yields strict performance gains over strong baselines. Extensive experiments across various general and multi-hop QA datasets demonstrate that Search-R2 consistently outperforms strong RAG and RL-based baselines across model scales, achieving superior reasoning accuracy with minimal overhead.
6982bfb79084cb4f0ecb570c
Bowei He
6982bfb79084cb4f0ecb570d
Minda Hu
6982bfb79084cb4f0ecb570e
Zenan Xu
6982bfb79084cb4f0ecb570f
Hongru Wang
6982bfb79084cb4f0ecb5710
Licheng Zong
6982bfb79084cb4f0ecb5711
Yankai Chen
6982bfb79084cb4f0ecb5712
Chen Ma
6982bfb79084cb4f0ecb5713
Xue Liu
6982bfb79084cb4f0ecb5714
Pluto Zhou
6982bfb79084cb4f0ecb5715
Irwin King
Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration
Search-integrated reasoning enables language agents to transcend static parametric knowledge by actively querying external sources. However, training these agents via reinforcement learning is hindered by the multi-scale credit assignment problem: existing methods typically rely on sparse, trajectory-level rewards that fail to distinguish between high-quality reasoning and fortuitous guesses, leading to redundant or misleading search behaviors. To address this, we propose Search-R2, a novel Actor-Refiner collaboration framework that enhances reasoning through targeted intervention, with both components jointly optimized during training. Our approach decomposes the generation process into an Actor, which produces initial reasoning trajectories, and a Meta-Refiner, which selectively diagnoses and repairs flawed steps via a 'cut-and-regenerate' mechanism. To provide fine-grained supervision, we introduce a hybrid reward design that couples outcome correctness with a dense process reward quantifying the information density of retrieved evidence. Theoretically, we formalize the Actor-Refiner interaction as a smoothed mixture policy, proving that selective correction yields strict performance gains over strong baselines. Extensive experiments across various general and multi-hop QA datasets demonstrate that Search-R2 consistently outperforms strong RAG and RL-based baselines across model scales, achieving superior reasoning accuracy with minimal overhead.
Search-R2 framework improves language agent reasoning through Actor-Refiner collaboration with targeted interventions and fine-grained reward supervision for better credit assignment in reinforcement learning.
reinforcement learning search-integrated reasoning Actor-Refiner collaboration multi-scale credit assignment trajectory-level rewards dense process reward hybrid reward design cut-and-regenerate mechanism smoothed mixture policy general QA multi-hop QA