VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training
Training stability remains a central challenge in reinforcement learning (RL) for large language models (LLMs). Policy staleness, asynchronous training, and mismatches between training and inference engines all cause the behavior policy to diverge from the current policy, risking training collapse. Importance sampling provides a principled correction for this distribution shift but suffers from high variance; existing remedies such as token-level clipping and sequence-level normalization lack a unified theoretical foundation. We propose Variational sEquence-level Soft Policy Optimization (VESPO). By incorporating variance reduction into a variational formulation over proposal distributions, VESPO derives a closed-form reshaping kernel that operates directly on sequence-level importance weights without length normalization. Experiments on mathematical reasoning benchmarks show that VESPO maintains stable training under staleness ratios up to 64x and fully asynchronous execution, and delivers consistent gains across both dense and Mixture-of-Experts models. Code is available at https://github.com/FloyedShen/VESPO
6992047b50fb2c0be47837f0
Guobin Shen
claimed_verified
2026-02-17T15:52:51.206Z
6475ff9b4c9fb8a4bf1cde76
/avatars/61cf82cd0e15c4618f5bd8b1f7d52f37.svg
floyed shen
floyed
user
floyed
6992047b50fb2c0be47837f1
Chenxiao Zhao
admin_assigned
2026-02-17T17:17:12.583Z
63fc5b724c57549ad5e54558
/avatars/1374c1e8969533dd7543959666f16d1a.svg
Chenxiao Zhao
ChenShawn
user
ChenShawn
6992047b50fb2c0be47837f2
Xiang Cheng
claimed_verified
2026-02-17T15:52:57.697Z
655c43d6b426ec8f4b5e7652
/avatars/ddcf9d1ef0e2dc1f564a56ba9153f24f.svg
Xiang Cheng
FFFc2
user
FFFc2
6992047b50fb2c0be47837f3
Lei Huang
claimed_verified
2026-02-23T09:46:50.954Z
61f4c2e981c4d30f58140279
/avatars/c4a69f6563c952354e33682e86045b14.svg
HuangMeow
Luckyyy
user
Luckyyy
6992047b50fb2c0be47837f4
Xing Yu
VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training
Training stability remains a central challenge in reinforcement learning (RL) for large language models (LLMs). Policy staleness, asynchronous training, and mismatches between training and inference engines all cause the behavior policy to diverge from the current policy, risking training collapse. Importance sampling provides a principled correction for this distribution shift but suffers from high variance; existing remedies such as token-level clipping and sequence-level normalization lack a unified theoretical foundation. We propose Variational sEquence-level Soft Policy Optimization (VESPO). By incorporating variance reduction into a variational formulation over proposal distributions, VESPO derives a closed-form reshaping kernel that operates directly on sequence-level importance weights without length normalization. Experiments on mathematical reasoning benchmarks show that VESPO maintains stable training under staleness ratios up to 64x and fully asynchronous execution, and delivers consistent gains across both dense and Mixture-of-Experts models. Code is available at https://github.com/FloyedShen/VESPO
VESPO addresses training instability in LLM reinforcement learning by using variational formulation with variance reduction to correct policy divergence without length normalization.
reinforcement learning large language models policy staleness asynchronous training importance sampling variance reduction variational formulation proposal distributions sequence-level importance weights mathematical reasoning benchmarks Mixture-of-Experts models