SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks
Proximal Policy Optimization (PPO) is central to aligning Large Language Models (LLMs) in reasoning tasks with verifiable rewards. However, standard token-level PPO struggles in this setting due to the instability of temporal credit assignment over long Chain-of-Thought (CoT) horizons and the prohibitive memory cost of the value model. While critic-free alternatives like GRPO mitigate these issues, they incur significant computational overhead by requiring multiple samples for baseline estimation, severely limiting training throughput. In this paper, we introduce Sequence-Level PPO (SPPO), a scalable algorithm that harmonizes the sample efficiency of PPO with the stability of outcome-based updates. SPPO reformulates the reasoning process as a Sequence-Level Contextual Bandit problem, employing a decoupled scalar value function to derive low-variance advantage signals without multi-sampling. Extensive experiments on mathematical benchmarks demonstrate that SPPO significantly surpasses standard PPO and matches the performance of computation-heavy group-based methods, offering a resource-efficient framework for aligning reasoning LLMs.
69dc873a2b6b8dc66ffeec5a
Tianyi Wang
claimed_verified
2026-04-15T12:08:31.395Z
6878e75c25d1ed7d2b56a36f
/avatars/5056dd5cdaa442b260bc2fd85eea133a.svg
TIANYI
BIMU233
user
BIMU233
69dc873a2b6b8dc66ffeec5b
Yixia Li
claimed_verified
2026-04-13T07:31:54.116Z
6645bdf6621ded608be9c37e
https://cdn-avatars.huggingface.co/v1/production/uploads/6645bdf6621ded608be9c37e/lHFUPBkqCyKc6KCaBUaCs.jpeg
Yixia Li
X1AOX1A
user
X1AOX1A
69dc873a2b6b8dc66ffeec5c
Long Li
69dc873a2b6b8dc66ffeec5d
Yibiao Chen
69dc873a2b6b8dc66ffeec5e
Shaohan Huang
69dc873a2b6b8dc66ffeec5f
Yun Chen
69dc873a2b6b8dc66ffeec60
Peng Li
69dc873a2b6b8dc66ffeec61
Yang Liu
69dc873a2b6b8dc66ffeec62
Guanhua Chen
SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks
Proximal Policy Optimization (PPO) is central to aligning Large Language Models (LLMs) in reasoning tasks with verifiable rewards. However, standard token-level PPO struggles in this setting due to the instability of temporal credit assignment over long Chain-of-Thought (CoT) horizons and the prohibitive memory cost of the value model. While critic-free alternatives like GRPO mitigate these issues, they incur significant computational overhead by requiring multiple samples for baseline estimation, severely limiting training throughput. In this paper, we introduce Sequence-Level PPO (SPPO), a scalable algorithm that harmonizes the sample efficiency of PPO with the stability of outcome-based updates. SPPO reformulates the reasoning process as a Sequence-Level Contextual Bandit problem, employing a decoupled scalar value function to derive low-variance advantage signals without multi-sampling. Extensive experiments on mathematical benchmarks demonstrate that SPPO significantly surpasses standard PPO and matches the performance of computation-heavy group-based methods, offering a resource-efficient framework for aligning reasoning LLMs.
Sequence-Level PPO addresses instability in long-chain-of-thought reasoning by reformulating the process as a contextual bandit problem with decoupled value functions for improved efficiency.
Proximal Policy Optimization Large Language Models Chain-of-Thought temporal credit assignment value model critic-free methods GRPO sequence-level contextual bandit decoupled scalar value function sample efficiency computational overhead mathematical benchmarks