Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs
Reinforcement learning (RL) has re-emerged as a natural approach for training interactive LLM agents in real-world environments. However, directly applying the widely used Group Relative Policy Optimization (GRPO) algorithm to multi-turn tasks exposes notable limitations, particularly in scenarios requiring long-horizon reasoning. To address these challenges, we investigate more stable and effective advantage estimation strategies, especially for multi-turn settings. We first explore Proximal Policy Optimization (PPO) as an alternative and find it to be more robust than GRPO. To further enhance PPO in multi-turn scenarios, we introduce turn-PPO, a variant that operates on a turn-level MDP formulation, as opposed to the commonly used token-level MDP. Our results on the WebShop and Sokoban datasets demonstrate the effectiveness of turn-PPO, both with and without long reasoning components.
6948bf7734f46eaf46cbb30e
Junbo Li
claimed_verified
2025-12-22T10:58:28.607Z
6317d681743def9b3612ebeb
/avatars/9940b35d5f8ae5df417bb020b609a8d0.svg
Junbo Li
ljb121002
user
ljb121002
6948bf7734f46eaf46cbb30f
Peng Zhou
6948bf7734f46eaf46cbb310
Rui Meng
6948bf7734f46eaf46cbb311
Meet P. Vadera
6948bf7734f46eaf46cbb312
Lihong Li
6948bf7734f46eaf46cbb313
Yang Li
Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs
Reinforcement learning (RL) has re-emerged as a natural approach for training interactive LLM agents in real-world environments. However, directly applying the widely used Group Relative Policy Optimization (GRPO) algorithm to multi-turn tasks exposes notable limitations, particularly in scenarios requiring long-horizon reasoning. To address these challenges, we investigate more stable and effective advantage estimation strategies, especially for multi-turn settings. We first explore Proximal Policy Optimization (PPO) as an alternative and find it to be more robust than GRPO. To further enhance PPO in multi-turn scenarios, we introduce turn-PPO, a variant that operates on a turn-level MDP formulation, as opposed to the commonly used token-level MDP. Our results on the WebShop and Sokoban datasets demonstrate the effectiveness of turn-PPO, both with and without long reasoning components.
Research investigates stable advantage estimation strategies for reinforcement learning in multi-turn tasks, finding PPO more robust than GRPO and introducing turn-PPO for improved performance in long-horizon reasoning scenarios.
Reinforcement learning Group Relative Policy Optimization Proximal Policy Optimization advantage estimation multi-turn tasks long-horizon reasoning turn-PPO WebShop Sokoban