Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learning
Large language model post-training relies on reinforcement learning to improve model capability and alignment quality. However, the off-policy training paradigm introduces distribution shift, which often pushes the policy beyond the trust region, leading to training instabilities manifested as fluctuations in policy entropy and unstable gradients. Although PPO-Clip mitigates this issue through importance clipping, it still overlooks the global distributional shift of actions. To address these challenges, we propose using the entropy ratio between the current and previous policies as a new global metric that effectively quantifies the relative change in policy exploration throughout updates. Building on this metric, we introduce an Entropy Ratio Clipping (ERC) mechanism that imposes bidirectional constraints on the entropy ratio. This stabilizes policy updates at the global distribution level and compensates for the inability of PPO-clip to regulate probability shifts of un-sampled actions. We integrate ERC into both DAPO and GPPO reinforcement learning algorithms. Experiments across multiple benchmarks show that ERC consistently improves performance.
6936451a3962c926cf6807d6
Zhenpeng Su
claimed_verified
2025-12-08T08:29:49.837Z
61c2cf8d1172fa7969904d99
https://cdn-avatars.huggingface.co/v1/production/uploads/61c2cf8d1172fa7969904d99/R10G5h3d9Q_YQ__Hc-H4k.jpeg
suu
Suu
user
Suu
6936451a3962c926cf6807d7
Leiyu Pan
6936451a3962c926cf6807d8
Minxuan Lv
6936451a3962c926cf6807d9
Tiehua Mei
6936451a3962c926cf6807da
Zijia Lin
6936451a3962c926cf6807db
Yuntao Li
6936451a3962c926cf6807dc
Wenping Hu
6936451a3962c926cf6807dd
Ruiming Tang
6936451a3962c926cf6807de
Kun Gai
6936451a3962c926cf6807df
Guorui Zhou
Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learning
Large language model post-training relies on reinforcement learning to improve model capability and alignment quality. However, the off-policy training paradigm introduces distribution shift, which often pushes the policy beyond the trust region, leading to training instabilities manifested as fluctuations in policy entropy and unstable gradients. Although PPO-Clip mitigates this issue through importance clipping, it still overlooks the global distributional shift of actions. To address these challenges, we propose using the entropy ratio between the current and previous policies as a new global metric that effectively quantifies the relative change in policy exploration throughout updates. Building on this metric, we introduce an Entropy Ratio Clipping (ERC) mechanism that imposes bidirectional constraints on the entropy ratio. This stabilizes policy updates at the global distribution level and compensates for the inability of PPO-clip to regulate probability shifts of un-sampled actions. We integrate ERC into both DAPO and GPPO reinforcement learning algorithms. Experiments across multiple benchmarks show that ERC consistently improves performance.
Entropy Ratio Clipping (ERC) mechanism stabilizes policy updates in reinforcement learning by addressing global distributional shifts, improving performance over existing methods.
reinforcement learning distribution shift trust region policy entropy unstable gradients PPO-Clip entropy ratio Entropy Ratio Clipping ERC DAPO GPPO