Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models
Generative Reward Models (GenRMs) and LLM-as-a-Judge exhibit deceptive alignment by producing correct judgments for incorrect reasons, as they are trained and evaluated to prioritize Outcome Accuracy, which undermines their ability to generalize during RLHF. We introduce Rationale Consistency, a fine-grained metric that quantifies the alignment between the model's reasoning process and human judgment. Our evaluation of frontier models reveals that rationale consistency effectively discriminates among state-of-the-art models and detects deceptive alignment, while outcome accuracy falls short in both respects. To mitigate this gap, we introduce a hybrid signal that combines rationale consistency with outcome accuracy for GenRM training. Our training method achieves state-of-the-art performance on RM-Bench (87.1%) and JudgeBench (82%), surpassing outcome-only baselines by an average of 5%. Using RM during RLHF, our method effectively improves performance as demonstrated on Arena Hard v2, notably yielding a 7% improvement in creative writing tasks. Further analysis confirms that our method escapes the deceptive alignment trap, effectively reversing the decline in rationale consistency observed in outcome-only training.
69844fe4e34659da7e1f50e6
Binghai Wang
69844fe4e34659da7e1f50e7
Yantao Liu
69844fe4e34659da7e1f50e8
Yuxuan Liu
69844fe4e34659da7e1f50e9
Tianyi Tang
69844fe4e34659da7e1f50ea
Shenzhi Wang
claimed_verified
2026-03-23T11:01:30.154Z
6486dde1f74857df3f1a5828
https://cdn-avatars.huggingface.co/v1/production/uploads/6486dde1f74857df3f1a5828/FgE80CpalBO5qqArdfwxA.jpeg
Shenzhi Wang
shenzhi-wang
user
shenzhi-wang
69844fe4e34659da7e1f50eb
Chang Gao
69844fe4e34659da7e1f50ec
Chujie Zheng
69844fe4e34659da7e1f50ed
Yichang Zhang
69844fe4e34659da7e1f50ee
Le Yu
69844fe4e34659da7e1f50ef
Shixuan Liu
69844fe4e34659da7e1f50f0
Tao Gui
69844fe4e34659da7e1f50f1
Qi Zhang
69844fe4e34659da7e1f50f2
Xuanjing Huang
69844fe4e34659da7e1f50f3
Bowen Yu
69844fe4e34659da7e1f50f4
Fei Huang
69844fe4e34659da7e1f50f5
Junyang Lin
Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models
Generative Reward Models (GenRMs) and LLM-as-a-Judge exhibit deceptive alignment by producing correct judgments for incorrect reasons, as they are trained and evaluated to prioritize Outcome Accuracy, which undermines their ability to generalize during RLHF. We introduce Rationale Consistency, a fine-grained metric that quantifies the alignment between the model's reasoning process and human judgment. Our evaluation of frontier models reveals that rationale consistency effectively discriminates among state-of-the-art models and detects deceptive alignment, while outcome accuracy falls short in both respects. To mitigate this gap, we introduce a hybrid signal that combines rationale consistency with outcome accuracy for GenRM training. Our training method achieves state-of-the-art performance on RM-Bench (87.1%) and JudgeBench (82%), surpassing outcome-only baselines by an average of 5%. Using RM during RLHF, our method effectively improves performance as demonstrated on Arena Hard v2, notably yielding a 7% improvement in creative writing tasks. Further analysis confirms that our method escapes the deceptive alignment trap, effectively reversing the decline in rationale consistency observed in outcome-only training.
Generative Reward Models suffer from deceptive alignment due to outcome accuracy prioritization, but rationale consistency metrics and hybrid training signals improve performance and generalization in RLHF.
Generative Reward Models LLM-as-a-Judge deceptive alignment outcome accuracy rationale consistency RLHF RM-Bench JudgeBench creative writing tasks