Length-Unbiased Sequence Policy Optimization: Revealing and Controlling Response Length Variation in RLVR
Recent applications of Reinforcement Learning with Verifiable Rewards (RLVR) to Large Language Models (LLMs) and Vision-Language Models (VLMs) have demonstrated significant success in enhancing reasoning capabilities for complex tasks. During RLVR training, an increase in response length is often regarded as a key factor contributing to the growth of reasoning ability. However, the patterns of change in response length vary significantly across different RLVR algorithms during the training process. To provide a fundamental explanation for these variations, this paper conducts an in-depth analysis of the components of mainstream RLVR algorithms. We present a theoretical analysis of the factors influencing response length and validate our theory through extensive experimentation. Building upon these theoretical findings, we propose the Length-Unbiased Sequence Policy Optimization (LUSPO) algorithm. Specifically, we rectify the length bias inherent in Group Sequence Policy Optimization (GSPO), rendering its loss function unbiased with respect to response length and thereby resolving the issue of response length collapse. We conduct extensive experiments across mathematical reasoning benchmarks and multimodal reasoning scenarios, where LUSPO consistently achieves superior performance. Empirical results demonstrate that LUSPO represents a novel, state-of-the-art optimization strategy compared to existing methods such as GRPO and GSPO.
69855cbe4ad556f294b7eb29
Fanfan Liu
claimed_verified
2026-02-06T18:51:59.637Z
6689f7a1683da3ea29b4cee5
/avatars/988641cde34a60d183208fd9a2a72392.svg
Fanfan Liu
liufanfanlff
user
liufanfanlff
69855cbe4ad556f294b7eb2a
Youyang Yin
69855cbe4ad556f294b7eb2b
Peng Shi
69855cbe4ad556f294b7eb2c
Siqi Yang
69855cbe4ad556f294b7eb2d
Zhixiong Zeng
69855cbe4ad556f294b7eb2e
Haibo Qiu
Length-Unbiased Sequence Policy Optimization: Revealing and Controlling Response Length Variation in RLVR
Recent applications of Reinforcement Learning with Verifiable Rewards (RLVR) to Large Language Models (LLMs) and Vision-Language Models (VLMs) have demonstrated significant success in enhancing reasoning capabilities for complex tasks. During RLVR training, an increase in response length is often regarded as a key factor contributing to the growth of reasoning ability. However, the patterns of change in response length vary significantly across different RLVR algorithms during the training process. To provide a fundamental explanation for these variations, this paper conducts an in-depth analysis of the components of mainstream RLVR algorithms. We present a theoretical analysis of the factors influencing response length and validate our theory through extensive experimentation. Building upon these theoretical findings, we propose the Length-Unbiased Sequence Policy Optimization (LUSPO) algorithm. Specifically, we rectify the length bias inherent in Group Sequence Policy Optimization (GSPO), rendering its loss function unbiased with respect to response length and thereby resolving the issue of response length collapse. We conduct extensive experiments across mathematical reasoning benchmarks and multimodal reasoning scenarios, where LUSPO consistently achieves superior performance. Empirical results demonstrate that LUSPO represents a novel, state-of-the-art optimization strategy compared to existing methods such as GRPO and GSPO.
Research analyzes RLVR algorithms' impact on response length in LLMs and VLMs, proposing LUSPO to eliminate length bias and improve reasoning performance.
Reinforcement Learning with Verifiable Rewards LLMs Vision-Language Models response length sequence policy optimization Group Sequence Policy Optimization Length-Unbiased Sequence Policy Optimization mathematical reasoning multimodal reasoning