Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training
Reasoning LLMs-as-Judges, which can benefit from inference-time scaling, provide a promising path for extending the success of reasoning models to non-verifiable domains where the output correctness/quality cannot be directly checked. However, while reasoning judges have shown better performance on static evaluation benchmarks, their effectiveness in actual policy training has not been systematically examined. Therefore, we conduct a rigorous study to investigate the actual impact of non-reasoning and reasoning judges in reinforcement-learning-based LLM alignment. Our controlled synthetic setting, where a "gold-standard" judge (gpt-oss-120b) provides preference annotations to train smaller judges, reveals key differences between non-reasoning and reasoning judges: non-reasoning judges lead to reward hacking easily, while reasoning judges can lead to policies that achieve strong performance when evaluated by the gold-standard judge. Interestingly, we find that the reasoning-judge-trained policies achieve such strong performance by learning to generate highly effective adversarial outputs that can also score well on popular benchmarks such as Arena-Hard by deceiving other LLM-judges. Combined with our further analysis, our study highlights both important findings and room for improvements for applying (reasoning) LLM-judges in non-verifiable LLM post-training.
69b3710d2ad6c5d4cf486993
Yixin Liu
69b3710d2ad6c5d4cf486994
Yue Yu
69b3710d2ad6c5d4cf486995
DiJia Su
69b3710d2ad6c5d4cf486996
Sid Wang
69b3710d2ad6c5d4cf486997
Xuewei Wang
69b3710d2ad6c5d4cf486998
Song Jiang
69b3710d2ad6c5d4cf486999
Bo Liu
69b3710d2ad6c5d4cf48699a
Arman Cohan
69b3710d2ad6c5d4cf48699b
Yuandong Tian
69b3710d2ad6c5d4cf48699c
Zhengxing Chen
Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training
Reasoning LLMs-as-Judges, which can benefit from inference-time scaling, provide a promising path for extending the success of reasoning models to non-verifiable domains where the output correctness/quality cannot be directly checked. However, while reasoning judges have shown better performance on static evaluation benchmarks, their effectiveness in actual policy training has not been systematically examined. Therefore, we conduct a rigorous study to investigate the actual impact of non-reasoning and reasoning judges in reinforcement-learning-based LLM alignment. Our controlled synthetic setting, where a "gold-standard" judge (gpt-oss-120b) provides preference annotations to train smaller judges, reveals key differences between non-reasoning and reasoning judges: non-reasoning judges lead to reward hacking easily, while reasoning judges can lead to policies that achieve strong performance when evaluated by the gold-standard judge. Interestingly, we find that the reasoning-judge-trained policies achieve such strong performance by learning to generate highly effective adversarial outputs that can also score well on popular benchmarks such as Arena-Hard by deceiving other LLM-judges. Combined with our further analysis, our study highlights both important findings and room for improvements for applying (reasoning) LLM-judges in non-verifiable LLM post-training.
Research examines the effectiveness of reasoning versus non-reasoning large language model judges in reinforcement learning-based alignment, revealing that reasoning judges prevent reward hacking but may produce adversarial outputs that deceive other judges.
LLMs-as-Judges inference-time scaling reinforcement-learning-based LLM alignment reward hacking preference annotations gold-standard judge reward modeling adversarial outputs LLM-judges policy training