Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
This paper proposes a novel formulation for reinforcement learning (RL) with large language models, explaining why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE. Specifically, through a first-order approximation, we show that this surrogate becomes increasingly valid only when both the training-inference discrepancy and policy staleness are minimized. This insight provides a principled explanation for the crucial role of several widely adopted techniques in stabilizing RL training, including importance sampling correction, clipping, and particularly Routing Replay for Mixture-of-Experts (MoE) models. Through extensive experiments with a 30B MoE model totaling hundreds of thousands of GPU hours, we show that for on-policy training, the basic policy gradient algorithm with importance sampling correction achieves the highest training stability. When off-policy updates are introduced to accelerate convergence, combining clipping and Routing Replay becomes essential to mitigate the instability caused by policy staleness. Notably, once training is stabilized, prolonged optimization consistently yields comparable final performance regardless of cold-start initialization. We hope that the shared insights and the developed recipes for stable RL training will facilitate future research.
692e6bf937312eaa83fd8890
Chujie Zheng
claimed_verified
2025-12-02T08:39:27.206Z
610b70452719facd4ea85e28
https://cdn-avatars.huggingface.co/v1/production/uploads/610b70452719facd4ea85e28/S7nMy7D0Rxq0VIVblhYDG.jpeg
Chujie Zheng
chujiezheng
user
chujiezheng
692e6bf937312eaa83fd8891
Kai Dang
692e6bf937312eaa83fd8892
Bowen Yu
692e6bf937312eaa83fd8893
Mingze Li
692e6bf937312eaa83fd8894
Huiqiang Jiang
claimed_verified
2025-12-03T09:17:41.367Z
6278bd42541f3d2dfa77ea70
https://cdn-avatars.huggingface.co/v1/production/uploads/6278bd42541f3d2dfa77ea70/ejn49eapnB3UXQckAYdTd.jpeg
Huiqiang Jiang
iofu728
user
iofu728
692e6bf937312eaa83fd8895
Junrong Lin
692e6bf937312eaa83fd8896
Yuqiong Liu
692e6bf937312eaa83fd8897
An Yang
claimed_verified
2025-12-02T08:39:25.208Z
62088594a5943c8a8fc94560
https://cdn-avatars.huggingface.co/v1/production/uploads/1644733028938-62088594a5943c8a8fc94560.png
An Yang
yangapku
user
yangapku
692e6bf937312eaa83fd8898
Jingren Zhou
692e6bf937312eaa83fd8899
Junyang Lin
Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
This paper proposes a novel formulation for reinforcement learning (RL) with large language models, explaining why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE. Specifically, through a first-order approximation, we show that this surrogate becomes increasingly valid only when both the training-inference discrepancy and policy staleness are minimized. This insight provides a principled explanation for the crucial role of several widely adopted techniques in stabilizing RL training, including importance sampling correction, clipping, and particularly Routing Replay for Mixture-of-Experts (MoE) models. Through extensive experiments with a 30B MoE model totaling hundreds of thousands of GPU hours, we show that for on-policy training, the basic policy gradient algorithm with importance sampling correction achieves the highest training stability. When off-policy updates are introduced to accelerate convergence, combining clipping and Routing Replay becomes essential to mitigate the instability caused by policy staleness. Notably, once training is stabilized, prolonged optimization consistently yields comparable final performance regardless of cold-start initialization. We hope that the shared insights and the developed recipes for stable RL training will facilitate future research.
The paper provides a theoretical foundation for optimizing sequence-level rewards in reinforcement learning using token-level objectives, highlighting the importance of techniques like importance sampling correction, clipping, and Routing Replay for stabilizing training, especially with large language models.
reinforcement learning large language models sequence-level reward token-level objective policy gradient methods REINFORCE first-order approximation training-inference discrepancy policy staleness importance sampling correction clipping Routing Replay Mixture-of-Experts on-policy training off-policy updates