BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning
Proximal constraints are fundamental to the stability of the Large Language Model reinforcement learning. While the canonical clipping mechanism in PPO serves as an efficient surrogate for trust regions, we identify a critical bottleneck: fixed bounds strictly constrain the upward update margin of low-probability actions, disproportionately suppressing high-advantage tail strategies and inducing rapid entropy collapse. To address this, we introduce Band-constrained Policy Optimization (BandPO). BandPO replaces canonical clipping with Band, a unified theoretical operator that projects trust regions defined by f-divergences into dynamic, probability-aware clipping intervals. Theoretical analysis confirms that Band effectively resolves this exploration bottleneck. We formulate this mapping as a convex optimization problem, guaranteeing a globally optimal numerical solution while deriving closed-form solutions for specific divergences. Extensive experiments across diverse models and datasets demonstrate that BandPO consistently outperforms canonical clipping and Clip-Higher, while robustly mitigating entropy collapse.
69aa4fe4c4f6edddcc6c86d3
Yuan Li
claimed_verified
2026-03-06T13:55:05.159Z
6833fc195b1c7d1451990ff5
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/b1CPqvruK5Us11AcpZb1T.png
Yuan-Li-FNLP
Yuan-Li-FNLP
user
Yuan-Li-FNLP
69aa4fe4c4f6edddcc6c86d4
Bo Wang
claimed_verified
2026-04-15T12:09:40.233Z
663c23b4a4152b77b67b7c1c
/avatars/7af03530ce2b48bc77ce84296878978f.svg
Bo Wang
Musicode
user
Musicode
69aa4fe4c4f6edddcc6c86d5
Yufei Gao
69aa4fe4c4f6edddcc6c86d6
Yuqian Yao
69aa4fe4c4f6edddcc6c86d7
Xinyuan Wang
claimed_verified
2026-03-06T13:55:07.361Z
63eb133a91a1b8ec4fbc4c2f
https://cdn-avatars.huggingface.co/v1/production/uploads/63eb133a91a1b8ec4fbc4c2f/dmaD56RAqkovB4izizv5m.png
Xinyuan Wang
buaa42wxy
user
buaa42wxy
69aa4fe4c4f6edddcc6c86d8
Zhangyue Yin
69aa4fe4c4f6edddcc6c86d9
Xipeng Qiu
BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning
Proximal constraints are fundamental to the stability of the Large Language Model reinforcement learning. While the canonical clipping mechanism in PPO serves as an efficient surrogate for trust regions, we identify a critical bottleneck: fixed bounds strictly constrain the upward update margin of low-probability actions, disproportionately suppressing high-advantage tail strategies and inducing rapid entropy collapse. To address this, we introduce Band-constrained Policy Optimization (BandPO). BandPO replaces canonical clipping with Band, a unified theoretical operator that projects trust regions defined by f-divergences into dynamic, probability-aware clipping intervals. Theoretical analysis confirms that Band effectively resolves this exploration bottleneck. We formulate this mapping as a convex optimization problem, guaranteeing a globally optimal numerical solution while deriving closed-form solutions for specific divergences. Extensive experiments across diverse models and datasets demonstrate that BandPO consistently outperforms canonical clipping and Clip-Higher, while robustly mitigating entropy collapse.
Band-constrained Policy Optimization addresses stability issues in reinforcement learning for large language models by replacing fixed clipping with a dynamic probability-aware projection method that prevents entropy collapse.
PPO trust regions f-divergences convex optimization entropy collapse Band-constrained Policy Optimization canonical clipping probability-aware clipping