Real-Time Aligned Reward Model beyond Semantics
Reinforcement Learning from Human Feedback (RLHF) is a pivotal technique for aligning large language models (LLMs) with human preferences, yet it is susceptible to reward overoptimization, in which policy models overfit to the reward model, exploit spurious reward patterns instead of faithfully capturing human intent. Prior mitigations primarily relies on surface semantic information and fails to efficiently address the misalignment between the reward model (RM) and the policy model caused by continuous policy distribution shifts. This inevitably leads to an increasing reward discrepancy, exacerbating reward overoptimization. To address these limitations, we introduce R2M (Real-Time Aligned Reward Model), a novel lightweight RLHF framework. R2M goes beyond vanilla reward models that solely depend on the semantic representations of a pretrained LLM. Instead, it leverages the evolving hidden states of the policy (namely policy feedback) to align with the real-time distribution shift of the policy during the RL process. This work points to a promising new direction for improving the performance of reward models through real-time utilization of feedback from policy models.
6980162d6676f9332270656f
Zixuan Huang
claimed_verified
2026-02-02T16:56:16.769Z
6593d2329e16fa7510e0876a
/avatars/c200676dd9dc1db8a3e27388251aea49.svg
hzx
hzxllll
user
hzxllll
6980162d6676f93322706570
Xin Xia
6980162d6676f93322706571
Yuxi Ren
6980162d6676f93322706572
Jianbin Zheng
6980162d6676f93322706573
Xuefeng Xiao
6980162d6676f93322706574
Hongyan Xie
6980162d6676f93322706575
Li Huaqiu
6980162d6676f93322706576
Songshi Liang
6980162d6676f93322706577
Zhongxiang Dai
6980162d6676f93322706578
Fuzhen Zhuang
6980162d6676f93322706579
Jianxin Li
6980162d6676f9332270657a
Yikun Ban
claimed_verified
2026-02-02T16:56:12.351Z
68345345f4bbf856e2d708e2
https://cdn-avatars.huggingface.co/v1/production/uploads/68345345f4bbf856e2d708e2/L5H2HNCuWje3ti2tNbC5p.jpeg
Yikun Ban
Yikunb
user
Yikunb
6980162d6676f9332270657b
Deqing Wang
Real-Time Aligned Reward Model beyond Semantics
Reinforcement Learning from Human Feedback (RLHF) is a pivotal technique for aligning large language models (LLMs) with human preferences, yet it is susceptible to reward overoptimization, in which policy models overfit to the reward model, exploit spurious reward patterns instead of faithfully capturing human intent. Prior mitigations primarily relies on surface semantic information and fails to efficiently address the misalignment between the reward model (RM) and the policy model caused by continuous policy distribution shifts. This inevitably leads to an increasing reward discrepancy, exacerbating reward overoptimization. To address these limitations, we introduce R2M (Real-Time Aligned Reward Model), a novel lightweight RLHF framework. R2M goes beyond vanilla reward models that solely depend on the semantic representations of a pretrained LLM. Instead, it leverages the evolving hidden states of the policy (namely policy feedback) to align with the real-time distribution shift of the policy during the RL process. This work points to a promising new direction for improving the performance of reward models through real-time utilization of feedback from policy models.
RLHF suffers from reward overoptimization due to misalignment between reward models and policy models, which R2M addresses by incorporating real-time policy feedback to dynamically adapt reward modeling during training.
Reinforcement Learning from Human Feedback reward overoptimization policy models reward models policy distribution shifts real-time alignment policy feedback