ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents
Multi-turn LLM agents are increasingly important for solving complex, interactive tasks, and reinforcement learning (RL) is a key ingredient for improving their long-horizon behavior. However, RL training requires generating large numbers of sandboxed rollout trajectories, and existing infrastructures often couple rollout orchestration with the training loop, making systems hard to migrate and maintain. Under the rollout-as-a-service philosophy, we present ProRL Agent , a scalable infrastructure that serves the full agentic rollout lifecycle through an API service. ProRL Agent also provides standardized and extensible sandbox environments that support diverse agentic tasks in rootless HPC settings. We validate ProRL Agent through RL training on software engineering, math, STEM, and coding tasks. ProRL Agent is open-sourced and integrated as part of NVIDIA NeMo Gym.
69bcb6be081b9ad49af57289
Hao Zhang
69bcb6be081b9ad49af5728a
Mingjie Liu
69bcb6be081b9ad49af5728b
Shaokun Zhang
69bcb6be081b9ad49af5728c
Songyang Han
69bcb6be081b9ad49af5728d
Jian Hu
69bcb6be081b9ad49af5728e
Zhenghui Jin
69bcb6be081b9ad49af5728f
Yuchi Zhang
69bcb6be081b9ad49af57290
Shizhe Diao
69bcb6be081b9ad49af57291
Ximing Lu
69bcb6be081b9ad49af57292
Binfeng Xu
69bcb6be081b9ad49af57293
Zhiding Yu
69bcb6be081b9ad49af57294
Jan Kautz
69bcb6be081b9ad49af57295
Yi Dong
ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents
Multi-turn LLM agents are increasingly important for solving complex, interactive tasks, and reinforcement learning (RL) is a key ingredient for improving their long-horizon behavior. However, RL training requires generating large numbers of sandboxed rollout trajectories, and existing infrastructures often couple rollout orchestration with the training loop, making systems hard to migrate and maintain. Under the rollout-as-a-service philosophy, we present ProRL Agent , a scalable infrastructure that serves the full agentic rollout lifecycle through an API service. ProRL Agent also provides standardized and extensible sandbox environments that support diverse agentic tasks in rootless HPC settings. We validate ProRL Agent through RL training on software engineering, math, STEM, and coding tasks. ProRL Agent is open-sourced and integrated as part of NVIDIA NeMo Gym.
Reinforcement learning infrastructure for multi-turn LLM agents that provides scalable rollout services and standardized sandbox environments for complex interactive tasks.
multi-turn LLM agents reinforcement learning rollout trajectories sandboxed rollout trajectories agentic rollout lifecycle API service sandbox environments rootless HPC settings software engineering math STEM coding tasks