Process Reward Agents for Steering Knowledge-Intensive Reasoning
Reasoning in knowledge-intensive domains remains challenging as intermediate steps are often not locally verifiable: unlike math or code, evaluating step correctness may require synthesizing clues across large external knowledge sources. As a result, subtle errors can propagate through reasoning traces, potentially never to be detected. Prior work has proposed process reward models (PRMs), including retrieval-augmented variants, but these methods operate post hoc, scoring completed trajectories, which prevents their integration into dynamic inference procedures. Here, we introduce Process Reward Agents (PRA), a test-time method for providing domain-grounded, online, step-wise rewards to a frozen policy. In contrast to prior retrieval-augmented PRMs, PRA enables search-based decoding to rank and prune candidate trajectories at every generation step. Experiments on multiple medical reasoning benchmarks demonstrate that PRA consistently outperforms strong baselines, achieving 80.8% accuracy on MedQA with Qwen3-4B, a new state of the art at the 4B scale. Importantly, PRA generalizes to unseen frozen policy models ranging from 0.5B to 8B parameters, improving their accuracy by up to 25.7% without any policy model updates. More broadly, PRA suggests a paradigm in which frozen reasoners are decoupled from domain-specific reward modules, allowing the deployment of new backbones in complex domains without retraining.
69dd0c7b65e6d8cd9cd10069
Jiwoong Sohn
claimed_verified
2026-04-13T19:23:40.937Z
64660e9ecf550af36eb2b774
https://cdn-avatars.huggingface.co/v1/production/uploads/64660e9ecf550af36eb2b774/NKFB8ltszcqPnr6BxhG6V.jpeg
Jiwoong Sohn
jw-sohn
user
jw-sohn
69dd0c7b65e6d8cd9cd1006a
Tomasz Sternal
69dd0c7b65e6d8cd9cd1006b
Kenneth Styppa
claimed_verified
2026-04-17T09:43:34.014Z
65a6a838c2cf664a51731850
https://cdn-avatars.huggingface.co/v1/production/uploads/65a6a838c2cf664a51731850/kzAxniypaNexpJ6TsyIz0.jpeg
Kenneth Styppa
KennySty
user
KennySty
69dd0c7b65e6d8cd9cd1006c
Torsten Hoefler
69dd0c7b65e6d8cd9cd1006d
Michael Moor
Process Reward Agents for Steering Knowledge-Intensive Reasoning
Reasoning in knowledge-intensive domains remains challenging as intermediate steps are often not locally verifiable: unlike math or code, evaluating step correctness may require synthesizing clues across large external knowledge sources. As a result, subtle errors can propagate through reasoning traces, potentially never to be detected. Prior work has proposed process reward models (PRMs), including retrieval-augmented variants, but these methods operate post hoc, scoring completed trajectories, which prevents their integration into dynamic inference procedures. Here, we introduce Process Reward Agents (PRA), a test-time method for providing domain-grounded, online, step-wise rewards to a frozen policy. In contrast to prior retrieval-augmented PRMs, PRA enables search-based decoding to rank and prune candidate trajectories at every generation step. Experiments on multiple medical reasoning benchmarks demonstrate that PRA consistently outperforms strong baselines, achieving 80.8% accuracy on MedQA with Qwen3-4B, a new state of the art at the 4B scale. Importantly, PRA generalizes to unseen frozen policy models ranging from 0.5B to 8B parameters, improving their accuracy by up to 25.7% without any policy model updates. More broadly, PRA suggests a paradigm in which frozen reasoners are decoupled from domain-specific reward modules, allowing the deployment of new backbones in complex domains without retraining.
Process Reward Agents provide domain-grounded, online step-wise rewards for frozen policies in knowledge-intensive reasoning, enabling improved search-based decoding and generalizing across different model sizes without retraining.
process reward models retrieval-augmented models test-time method frozen policy search-based decoding medical reasoning benchmarks MedQA Qwen3-4B parameter-efficient fine-tuning domain-specific reward modules