A Unified Framework for Rethinking Policy Divergence Measures in GRPO
Reinforcement Learning with Verified Reward (RLVR) has emerged as a critical paradigm for advancing the reasoning capabilities of Large Language Models (LLMs). Most existing RLVR methods, such as GRPO and its variants, ensure stable updates by constraining policy divergence through clipping likelihood ratios. This paper introduces a unified clipping framework that characterizes existing methods via a general notion of policy divergence, encompassing both likelihood ratios and Kullback-Leibler (KL) divergences and extending to alternative measures. The framework provides a principled foundation for systematically analyzing how different policy divergence measures affect exploration and performance. We further identify the KL3 estimator, a variance-reduced Monte Carlo estimator of the KL divergence, as a key policy divergence constraint. We theoretically demonstrate that the KL3-based constraint is mathematically equivalent to an asymmetric ratio-based clipping that reallocates probability mass toward high-confidence actions, promoting stronger exploration while retaining the simplicity of GRPO-style methods. Empirical results on mathematical reasoning benchmarks demonstrate that incorporating the KL3 estimator into GRPO improves both training stability and final performance, highlighting the importance of principled policy divergence constraints in policy optimization.
6985c2a1e387e657cdfcbce8
Qingyuan Wu
claimed_verified
2026-02-09T08:35:29.791Z
65d61e7888196f05410f3aa3
https://cdn-avatars.huggingface.co/v1/production/uploads/65d61e7888196f05410f3aa3/E7COFUfyzVo37aOJ_18pR.jpeg
qy
qingyuanwu
user
qingyuanwu
6985c2a1e387e657cdfcbce9
Yuhui Wang
6985c2a1e387e657cdfcbcea
Simon Sinong Zhan
6985c2a1e387e657cdfcbceb
Yanning Dai
6985c2a1e387e657cdfcbcec
Shilong Deng
claimed_verified
2026-02-06T18:50:03.240Z
63dd3f4ea8877129a15a66d1
/avatars/5410233517f4bcfae827094dc1fb7bcf.svg
SHILONG DENG
zczlsde
user
zczlsde
6985c2a1e387e657cdfcbced
Sarra Habchi
6985c2a1e387e657cdfcbcee
Qi Zhu
6985c2a1e387e657cdfcbcef
Matthias Gallé
6985c2a1e387e657cdfcbcf0
Chao Huang
A Unified Framework for Rethinking Policy Divergence Measures in GRPO
Reinforcement Learning with Verified Reward (RLVR) has emerged as a critical paradigm for advancing the reasoning capabilities of Large Language Models (LLMs). Most existing RLVR methods, such as GRPO and its variants, ensure stable updates by constraining policy divergence through clipping likelihood ratios. This paper introduces a unified clipping framework that characterizes existing methods via a general notion of policy divergence, encompassing both likelihood ratios and Kullback-Leibler (KL) divergences and extending to alternative measures. The framework provides a principled foundation for systematically analyzing how different policy divergence measures affect exploration and performance. We further identify the KL3 estimator, a variance-reduced Monte Carlo estimator of the KL divergence, as a key policy divergence constraint. We theoretically demonstrate that the KL3-based constraint is mathematically equivalent to an asymmetric ratio-based clipping that reallocates probability mass toward high-confidence actions, promoting stronger exploration while retaining the simplicity of GRPO-style methods. Empirical results on mathematical reasoning benchmarks demonstrate that incorporating the KL3 estimator into GRPO improves both training stability and final performance, highlighting the importance of principled policy divergence constraints in policy optimization.
A unified framework for reinforcement learning with verified reward is presented, characterized by policy divergence measures including likelihood ratios and KL divergences, with empirical validation showing improved training stability and performance through the KL3 estimator.
Reinforcement Learning with Verified Reward GRPO policy divergence likelihood ratios Kullback-Leibler divergence KL3 estimator variance-reduced Monte Carlo estimator policy optimization mathematical reasoning benchmarks training stability final performance