Clipping-Free Policy Optimization for Large Language Models
Reinforcement learning has become central to post-training large language models, yet dominant algorithms rely on clipping mechanisms that introduce optimization issues at scale, including zero-gradient regions, reward hacking, and training instability. We propose Clipping-Free Policy Optimization (CFPO), which replaces heuristic clipping with a convex quadratic penalty derived from Total Variation divergence constraints, yielding an everywhere-differentiable objective that enforces stable policy updates without hard boundaries. We evaluate CFPO across both reasoning and alignment settings. In reasoning, CFPO matches clipping-based methods on downstream benchmarks while extending the stable training regime. In alignment, CFPO mitigates verbosity exploitation and reduces capability degradation, while achieving competitive instruction-following performance. CFPO requires only a one-line code change and no additional hyperparameters. Our results suggest that CFPO is a promising drop-in alternative to clipping-based methods for LLM post-training.
698229cd9c2f139721ec3527
Ömer Veysel Çağatan
698229cd9c2f139721ec3528
Barış Akgün
698229cd9c2f139721ec3529
Gözde Gül Şahin
698229cd9c2f139721ec352a
Xuandong Zhao
claimed_verified
2026-02-04T12:30:17.048Z
6275a465597c70eb8949fce5
https://cdn-avatars.huggingface.co/v1/production/uploads/6275a465597c70eb8949fce5/ph4UogqMurMB0hSXZC38w.png
Xuandong Zhao
Xuandong
user
Xuandong
Clipping-Free Policy Optimization for Large Language Models
Reinforcement learning has become central to post-training large language models, yet dominant algorithms rely on clipping mechanisms that introduce optimization issues at scale, including zero-gradient regions, reward hacking, and training instability. We propose Clipping-Free Policy Optimization (CFPO), which replaces heuristic clipping with a convex quadratic penalty derived from Total Variation divergence constraints, yielding an everywhere-differentiable objective that enforces stable policy updates without hard boundaries. We evaluate CFPO across both reasoning and alignment settings. In reasoning, CFPO matches clipping-based methods on downstream benchmarks while extending the stable training regime. In alignment, CFPO mitigates verbosity exploitation and reduces capability degradation, while achieving competitive instruction-following performance. CFPO requires only a one-line code change and no additional hyperparameters. Our results suggest that CFPO is a promising drop-in alternative to clipping-based methods for LLM post-training.
Clipping-Free Policy Optimization replaces heuristic clipping with convex quadratic penalty to stabilize reinforcement learning training for large language models without performance loss.
reinforcement learning policy optimization clipping mechanisms optimization issues reward hacking training instability convex quadratic penalty Total Variation divergence policy updates instruction-following