Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training
Standard reward models typically predict scalar scores that fail to capture the multifaceted nature of response quality in non-verifiable domains, such as creative writing or open-ended instruction following. To address this limitation, we propose Rubric-ARM, a framework that jointly optimizes a rubric generator and a judge using reinforcement learning from preference feedback. Unlike existing methods that rely on static rubrics or disjoint training pipelines, our approach treats rubric generation as a latent action learned to maximize judgment accuracy. We introduce an alternating optimization strategy to mitigate the non-stationarity of simultaneous updates, providing theoretical analysis that demonstrates how this schedule reduces gradient variance during training. Extensive experiments show that Rubric-ARM achieves state-of-the-art performance among baselines on multiple benchmarks and significantly improves downstream policy alignment in both offline and online reinforcement learning settings.
698183a2ce18b18628096306
Ran Xu
claimed_verified
2026-02-04T12:32:04.129Z
6621dd7623518bf2898788da
/avatars/837aca71471a19ea80ae55d619781c5a.svg
Ran Xu
ritaranx
user
ritaranx
698183a2ce18b18628096307
Tianci Liu
claimed_verified
2026-02-04T12:32:08.667Z
64bf811d76a6e2efcceabc00
https://cdn-avatars.huggingface.co/v1/production/uploads/64bf811d76a6e2efcceabc00/0p3zSIVqzoME25Zmfh7SD.png
Tianci Liu
lliutianc
user
lliutianc
698183a2ce18b18628096308
Zihan Dong
698183a2ce18b18628096309
Tony You
698183a2ce18b1862809630a
Ilgee Hong
698183a2ce18b1862809630b
Carl Yang
698183a2ce18b1862809630c
Linjun Zhang
698183a2ce18b1862809630d
Tao Zhao
698183a2ce18b1862809630e
Haoyu Wang
claimed_verified
2026-02-03T10:03:47.466Z
641a92bc4182690729c9324b
/avatars/f5d3de7f04fe77d0cfced51b5431c114.svg
haoyu wang
haoyuw
user
haoyuw
Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training
Standard reward models typically predict scalar scores that fail to capture the multifaceted nature of response quality in non-verifiable domains, such as creative writing or open-ended instruction following. To address this limitation, we propose Rubric-ARM, a framework that jointly optimizes a rubric generator and a judge using reinforcement learning from preference feedback. Unlike existing methods that rely on static rubrics or disjoint training pipelines, our approach treats rubric generation as a latent action learned to maximize judgment accuracy. We introduce an alternating optimization strategy to mitigate the non-stationarity of simultaneous updates, providing theoretical analysis that demonstrates how this schedule reduces gradient variance during training. Extensive experiments show that Rubric-ARM achieves state-of-the-art performance among baselines on multiple benchmarks and significantly improves downstream policy alignment in both offline and online reinforcement learning settings.
Rubric-ARM framework jointly optimizes rubric generation and judging through reinforcement learning to improve response quality assessment in creative and open-ended tasks.
reward models reinforcement learning preference feedback rubric generator judge alternating optimization non-stationarity gradient variance policy alignment offline reinforcement learning online reinforcement learning