ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning
Agentic reinforcement learning (ARL) has rapidly gained attention as a promising paradigm for training agents to solve complex, multi-step interactive tasks. Despite encouraging early results, ARL remains highly unstable, often leading to training collapse. This instability limits scalability to larger environments and longer interaction horizons, and constrains systematic exploration of algorithmic design choices. In this paper, we first propose ARLArena, a stable training recipe and systematic analysis framework that examines training stability in a controlled and reproducible setting. ARLArena first constructs a clean and standardized testbed. Then, we decompose policy gradient into four core design dimensions and assess the performance and stability of each dimension. Through this fine-grained analysis, we distill a unified perspective on ARL and propose SAMPO, a stable agentic policy optimization method designed to mitigate the dominant sources of instability in ARL. Empirically, SAMPO achieves consistently stable training and strong performance across diverse agentic tasks. Overall, this study provides a unifying policy gradient perspective for ARL and offers practical guidance for building stable and reproducible LLM-based agent training pipelines.
699fbaf1ebfce7fbcca91b4d
Xiaoxuan Wang
699fbaf1ebfce7fbcca91b4e
Han Zhang
699fbaf1ebfce7fbcca91b4f
Haixin Wang
claimed_verified
2026-02-27T16:43:17.564Z
643ecc400d1194da249b6014
https://cdn-avatars.huggingface.co/v1/production/uploads/643ecc400d1194da249b6014/uTKL7nIIOKh2kI5wX10BT.jpeg
UCLA_WHX
willhx
user
willhx
699fbaf1ebfce7fbcca91b50
Yidan Shi
699fbaf1ebfce7fbcca91b51
Ruoyan Li
699fbaf1ebfce7fbcca91b52
Kaiqiao Han
699fbaf1ebfce7fbcca91b53
Chenyi Tong
699fbaf1ebfce7fbcca91b54
Haoran Deng
699fbaf1ebfce7fbcca91b55
Renliang Sun
699fbaf1ebfce7fbcca91b56
Alexander Taylor
699fbaf1ebfce7fbcca91b57
Yanqiao Zhu
699fbaf1ebfce7fbcca91b58
Jason Cong
699fbaf1ebfce7fbcca91b59
Yizhou Sun
699fbaf1ebfce7fbcca91b5a
Wei Wang
ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning
Agentic reinforcement learning (ARL) has rapidly gained attention as a promising paradigm for training agents to solve complex, multi-step interactive tasks. Despite encouraging early results, ARL remains highly unstable, often leading to training collapse. This instability limits scalability to larger environments and longer interaction horizons, and constrains systematic exploration of algorithmic design choices. In this paper, we first propose ARLArena, a stable training recipe and systematic analysis framework that examines training stability in a controlled and reproducible setting. ARLArena first constructs a clean and standardized testbed. Then, we decompose policy gradient into four core design dimensions and assess the performance and stability of each dimension. Through this fine-grained analysis, we distill a unified perspective on ARL and propose SAMPO, a stable agentic policy optimization method designed to mitigate the dominant sources of instability in ARL. Empirically, SAMPO achieves consistently stable training and strong performance across diverse agentic tasks. Overall, this study provides a unifying policy gradient perspective for ARL and offers practical guidance for building stable and reproducible LLM-based agent training pipelines.
ARLArena framework analyzes training stability in agentic reinforcement learning and proposes SAMPO method for stable policy optimization across diverse tasks.
agentic reinforcement learning policy gradient training stability policy optimization ARLArena SAMPO