ECHO-2: A Large Scale Distributed Rollout Framework for Cost-efficient Reinforcement Learning

TL;DR

This work tackles the high cost of RL post-training for large language models by distributing rollout generation to wide-area inference workers while keeping a centralized learner. ECHO-2 introduces a three-plane, versioned execution framework with bounded staleness $S$ and an overlap-based capacity model that relates $T_{ ext{train}}$, $T_{ ext{bcast}}$, and rollout throughput to sustain learner utilization, complemented by a peer-assisted pipelined broadcast to reduce dissemination tail latency. Its three main contributions are (i) the three-plane architecture with bounded staleness and overlap-driven provisioning, (ii) a peer-assisted broadcast mechanism to minimize dissemination delays, and (iii) a cost-aware scheduling policy over heterogeneous resources. Empirically, ECHO-2 achieves substantial cost reductions (33–36% lower cost to reach target RL quality) for GRPO post-training on 4B and 8B models under realistic WAN regimes, while preserving comparable RL rewards to strong baselines. This framework offers a practical path to scalable, cost-efficient RL post-training across distributed resources, widening accessibility for large-scale RL workloads.

Abstract

Reinforcement learning (RL) is a critical stage in post-training large language models (LLMs), involving repeated interaction between rollout generation, reward evaluation, and centralized learning. Distributing rollout execution offers opportunities to leverage more cost-efficient inference resources, but introduces challenges in wide-area coordination and policy dissemination. We present ECHO-2, a distributed RL framework for post-training with remote inference workers and non-negligible dissemination latency. ECHO-2 combines centralized learning with distributed rollouts and treats bounded policy staleness as a user-controlled parameter, enabling rollout generation, dissemination, and training to overlap. We introduce an overlap-based capacity model that relates training time, dissemination latency, and rollout throughput, yielding a practical provisioning rule for sustaining learner utilization. To mitigate dissemination bottlenecks and lower cost, ECHO-2 employs peer-assisted pipelined broadcast and cost-aware activation of heterogeneous workers. Experiments on GRPO post-training of 4B and 8B models under real wide-area bandwidth regimes show that ECHO-2 significantly improves cost efficiency while preserving RL reward comparable to strong baselines.

Figures (7)

• Figure 1: Asynchronous RL execution in ECHO-2 with maximum bounded staleness $S=3$ and publication period $\kappa=2$. The rollout, generation, and learner update proceed concurrently. Rollout workers uses the latest policy snapshot to generate trajectories into the replay buffer. The learner consumes trajectories from replay buffer and broadcasts a new version of policy to rollout workers in each $\kappa$ training steps. ECHO-2 generates rollouts at a higher rate than it consumes during training.
• Figure 2: System Architecture of ECHO-2. The system adopts a three-plane decomposition for cost-efficient distributed RL. The centralized Learning Plane performs policy optimization using data sampled with a bounded staleness budget. The Data Plane provides a unified interface for task adaptation and manages versioned trajectory storage. The distributed Rollout Plane executes asynchronous generation across workers using pipelined broadcast.
• Figure 3: Experimental results of ECHO-2 on Qwen3-8B. (a) Cost–quality efficiency on AIME24 under the WAN setting. Dashed lines indicate computed costs based on steady-state training time and public GPU rental prices (right y-axis). (b) Impact of staleness $S$ on RL stability. Performance remains robust for $S \le 6$, while excessive staleness ($S=11$) leads to divergence. (c) Learner bubble ratio as a function of the number of rollout workers. Vertical dashed lines denote the theoretical minimum workers.
• Figure 4: Policy broadcast latency $T_{\text{bcast}}$ vs. rollout fleet size. Comparison of three dissemination strategies across different numbers of nodes $N$. Star-Limited (with learner uplink $B_0 \in [300, 800]\text{Mbps}$) suffers from linear latency growth as the learner becomes a bandwidth bottleneck. Tree-Pipelined dissemination, by utilizing chunked peer forwarding, maintains a near-constant broadcast time that scales efficiently with the fleet size, closely matching the idealized Star-Unlimited baseline.
• Figure 5: Cost--quality on AIME for Qwen3-4B.