ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning
Reinforcement learning (RL) is a critical stage in post-training large language models (LLMs), involving repeated interaction between rollout generation, reward evaluation, and centralized learning. Distributing rollout execution offers opportunities to leverage more cost-efficient inference resources, but introduces challenges in wide-area coordination and policy dissemination. We present ECHO-2, a distributed RL framework for post-training with remote inference workers and non-negligible dissemination latency. ECHO-2 combines centralized learning with distributed rollouts and treats bounded policy staleness as a user-controlled parameter, enabling rollout generation, dissemination, and training to overlap. We introduce an overlap-based capacity model that relates training time, dissemination latency, and rollout throughput, yielding a practical provisioning rule for sustaining learner utilization. To mitigate dissemination bottlenecks and lower cost, ECHO-2 employs peer-assisted pipelined broadcast and cost-aware activation of heterogeneous workers. Experiments on GRPO post-training of 4B and 8B models under real wide-area bandwidth regimes show that ECHO-2 significantly improves cost efficiency while preserving RL reward comparable to strong baselines.
698b4deb6052d3bed96307aa
Jie Xiao
698b4deb6052d3bed96307ab
Meng Chen
698b4deb6052d3bed96307ac
Qingnan Ren
698c9e456c5152984e4f3c98
Jingwei Song
698b4deb6052d3bed96307ae
Jiaqi Huang
698b4deb6052d3bed96307af
Yangshen Deng
698b4deb6052d3bed96307b0
Chris Tong
698b4deb6052d3bed96307b1
Wanyi Chen
698b4deb6052d3bed96307b2
Suli Wang
698b4deb6052d3bed96307b3
Ziqian Bi
698b4deb6052d3bed96307b4
Shuo Lu
698b4deb6052d3bed96307b5
Yiqun Duan
698b4deb6052d3bed96307b6
Xu Wang
698b4deb6052d3bed96307b7
Rymon Yu
698b4deb6052d3bed96307b8
Ween Yang
698b4deb6052d3bed96307b9
Lynn Ai
698b4deb6052d3bed96307ba
Eric Yang
698b4deb6052d3bed96307bb
Bill Shi
698b4deb6052d3bed96307ad
Song Jingwei
claimed_verified
2026-02-11T11:16:34.503Z
6703c272c265fe2565f90e5c
https://cdn-avatars.huggingface.co/v1/production/uploads/6703c272c265fe2565f90e5c/Fu3--OfA6tFYeASPrxc_c.png
Jingwei Song
JingweiSong
user
JingweiSong
ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning
Reinforcement learning (RL) is a critical stage in post-training large language models (LLMs), involving repeated interaction between rollout generation, reward evaluation, and centralized learning. Distributing rollout execution offers opportunities to leverage more cost-efficient inference resources, but introduces challenges in wide-area coordination and policy dissemination. We present ECHO-2, a distributed RL framework for post-training with remote inference workers and non-negligible dissemination latency. ECHO-2 combines centralized learning with distributed rollouts and treats bounded policy staleness as a user-controlled parameter, enabling rollout generation, dissemination, and training to overlap. We introduce an overlap-based capacity model that relates training time, dissemination latency, and rollout throughput, yielding a practical provisioning rule for sustaining learner utilization. To mitigate dissemination bottlenecks and lower cost, ECHO-2 employs peer-assisted pipelined broadcast and cost-aware activation of heterogeneous workers. Experiments on GRPO post-training of 4B and 8B models under real wide-area bandwidth regimes show that ECHO-2 significantly improves cost efficiency while preserving RL reward comparable to strong baselines.
ECHO-2 is a distributed reinforcement learning framework that enables efficient post-training of large language models by overlapping rollout generation, dissemination, and training while managing policy staleness and network latency.
reinforcement learning post-training large language models distributed RL rollout generation reward evaluation centralized learning policy staleness peer-assisted pipelined broadcast cost-aware activation GRPO