Your Group-Relative Advantage Is Biased
Reinforcement Learning from Verifier Rewards (RLVR) has emerged as a widely used approach for post-training large language models on reasoning tasks, with group-based methods such as GRPO and its variants gaining broad adoption. These methods rely on group-relative advantage estimation to avoid learned critics, yet its theoretical properties remain poorly understood.
  In this work, we uncover a fundamental issue of group-based RL: the group-relative advantage estimator is inherently biased relative to the true (expected) advantage. We provide the first theoretical analysis showing that it systematically underestimates advantages for hard prompts and overestimates them for easy prompts, leading to imbalanced exploration and exploitation. To address this issue, we propose History-Aware Adaptive Difficulty Weighting (HA-DW), an adaptive reweighting scheme that adjusts advantage estimates based on an evolving difficulty anchor and training dynamics. Both theoretical analysis and experiments on five mathematical reasoning benchmarks demonstrate that HA-DW consistently improves performance when integrated into GRPO and its variants. Our results suggest that correcting biased advantage estimation is critical for robust and efficient RLVR training.
69674059c5e371f6b235d1d8
Fengkai Yang
admin_assigned
2026-01-19T14:33:21.899Z
68920f91bcf2b25e8e121cf6
/avatars/4bc69f43828a346a3ee24b026e0edbb4.svg
Fengkai Yang
ShortCatisLong
user
ShortCatisLong
69674059c5e371f6b235d1d9
Zherui Chen
claimed_verified
2026-01-16T10:33:53.078Z
6969715fb2636f5f23a9a8c5
/avatars/5e75043891ee41bb980f71fb9e3a33ab.svg
Zherui Chen
chenzherui007
user
chenzherui007
69674059c5e371f6b235d1da
Xiaohan Wang
69674059c5e371f6b235d1db
Xiaodong Lu
69674059c5e371f6b235d1dc
Jiajun Chai
admin_assigned
2026-01-19T14:37:17.404Z
666eb642a119281ee0bfa443
/avatars/71317810b00978754ad439837b04faff.svg
Jiajun Chai
PandaChai
user
PandaChai
69674059c5e371f6b235d1dd
Guojun Yin
69674059c5e371f6b235d1de
Wei Lin
69674059c5e371f6b235d1df
Shuai Ma
69674059c5e371f6b235d1e0
Fuzhen Zhuang
69674059c5e371f6b235d1e1
Deqing Wang
69674059c5e371f6b235d1e2
Yaodong Yang
claimed_verified
2026-03-06T13:58:11.727Z
686894eb283604c2808f7c81
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/BGuMtSS3BGNicQiojujUO.png
yd yang
yyang13
user
yyang13
69674059c5e371f6b235d1e3
Jianxin Li
69674059c5e371f6b235d1e4
Yikun Ban
claimed_verified
2026-01-16T10:33:50.655Z
68345345f4bbf856e2d708e2
https://cdn-avatars.huggingface.co/v1/production/uploads/68345345f4bbf856e2d708e2/L5H2HNCuWje3ti2tNbC5p.jpeg
Yikun Ban
Yikunb
user
Yikunb
Your Group-Relative Advantage Is Biased
Reinforcement Learning from Verifier Rewards (RLVR) has emerged as a widely used approach for post-training large language models on reasoning tasks, with group-based methods such as GRPO and its variants gaining broad adoption. These methods rely on group-relative advantage estimation to avoid learned critics, yet its theoretical properties remain poorly understood.
  In this work, we uncover a fundamental issue of group-based RL: the group-relative advantage estimator is inherently biased relative to the true (expected) advantage. We provide the first theoretical analysis showing that it systematically underestimates advantages for hard prompts and overestimates them for easy prompts, leading to imbalanced exploration and exploitation. To address this issue, we propose History-Aware Adaptive Difficulty Weighting (HA-DW), an adaptive reweighting scheme that adjusts advantage estimates based on an evolving difficulty anchor and training dynamics. Both theoretical analysis and experiments on five mathematical reasoning benchmarks demonstrate that HA-DW consistently improves performance when integrated into GRPO and its variants. Our results suggest that correcting biased advantage estimation is critical for robust and efficient RLVR training.
Group-based reinforcement learning from verifier rewards suffers from biased advantage estimation that underestimates hard prompts and overestimates easy prompts, which is addressed through a history-aware adaptive difficulty weighting method that improves performance on mathematical reasoning benchmarks.
Reinforcement Learning from Verifier Rewards group-based methods GRPO advantage estimation bias correction adaptive reweighting difficulty weighting mathematical reasoning benchmark evaluation