Rethinking the Trust Region in LLM Reinforcement Learning
Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm. Despite its ubiquity, we argue that the core ratio clipping mechanism in PPO is structurally ill-suited for the large vocabularies inherent to LLMs. PPO constrains policy updates based on the probability ratio of sampled tokens, which serves as a noisy single-sample Monte Carlo estimate of the true policy divergence. This creates a sub-optimal learning dynamic: updates to low-probability tokens are aggressively over-penalized, while potentially catastrophic shifts in high-probability tokens are under-constrained, leading to training inefficiency and instability. To address this, we propose Divergence Proximal Policy Optimization (DPPO), which substitutes heuristic clipping with a more principled constraint based on a direct estimate of policy divergence (e.g., Total Variation or KL). To avoid huge memory footprint, we introduce the efficient Binary and Top-K approximations to capture the essential divergence with negligible overhead. Extensive empirical evaluations demonstrate that DPPO achieves superior training stability and efficiency compared to existing methods, offering a more robust foundation for RL-based LLM fine-tuning.
69840d8ae34659da7e1f4dda
Penghui Qi
claimed_verified
2026-02-05T10:54:11.972Z
63885f1d0bebb233d8ad6e5b
https://cdn-avatars.huggingface.co/v1/production/uploads/1669881620925-noauth.jpeg
Penghui Qi
QPHutu
user
QPHutu
69840d8ae34659da7e1f4ddb
Xiangxin Zhou
claimed_verified
2026-02-05T10:54:14.108Z
66129c7b50350afe76757262
/avatars/a2f4fac076b9d658a0d904ed54960f6f.svg
Xiangxin Zhou
zhouxiangxin
user
zhouxiangxin
69840d8ae34659da7e1f4ddc
Zichen Liu
69840d8ae34659da7e1f4ddd
Tianyu Pang
69840d8ae34659da7e1f4dde
Chao Du
69840d8ae34659da7e1f4ddf
Min Lin
69840d8ae34659da7e1f4de0
Wee Sun Lee
Rethinking the Trust Region in LLM Reinforcement Learning
Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm. Despite its ubiquity, we argue that the core ratio clipping mechanism in PPO is structurally ill-suited for the large vocabularies inherent to LLMs. PPO constrains policy updates based on the probability ratio of sampled tokens, which serves as a noisy single-sample Monte Carlo estimate of the true policy divergence. This creates a sub-optimal learning dynamic: updates to low-probability tokens are aggressively over-penalized, while potentially catastrophic shifts in high-probability tokens are under-constrained, leading to training inefficiency and instability. To address this, we propose Divergence Proximal Policy Optimization (DPPO), which substitutes heuristic clipping with a more principled constraint based on a direct estimate of policy divergence (e.g., Total Variation or KL). To avoid huge memory footprint, we introduce the efficient Binary and Top-K approximations to capture the essential divergence with negligible overhead. Extensive empirical evaluations demonstrate that DPPO achieves superior training stability and efficiency compared to existing methods, offering a more robust foundation for RL-based LLM fine-tuning.
DPPO addresses limitations in PPO for LLM fine-tuning by replacing ratio clipping with direct policy divergence constraints, improving training stability and efficiency.
Proximal Policy Optimization policy divergence policy updates token probability ratios Monte Carlo estimates reinforcement learning Large Language Models Total Variation KL divergence binary approximation Top-K approximation