Unleashing Efficient Asynchronous RL Post-Training via Staleness-Constrained Rollout Coordination

TL;DR

This work tackles RL post-training in fully disaggregated architectures by addressing two data-level bottlenecks: data staleness and data skewness. It introduces StaleFlow, which combines a global consistency protocol with a virtual staleness buffer, and new data planes (trajectory server and parameter server) plus a centralized coordinator to enable staleness-aware, throughput-oriented rollout coordination. The approach yields substantial throughput gains (up to $2.68\times$) while preserving convergence under bounded staleness, validated on large GPU clusters with diverse model families. This framework enables scalable, efficient RL post-training with rigorous data governance, offering practical impact for deploying high-performing models in asynchronous pipelines.

Abstract

Reinforcement learning (RL) post-training has become pivotal for enhancing the capabilities of modern large models. A recent trend is to develop RL systems with a fully disaggregated architecture, which decouples the three RL phases (rollout, reward, and training) onto separate resources and executes them asynchronously. However, two critical data-level concerns arise: (1) asynchronous execution leads to data staleness in trajectories (the data generated by rollout) as the model parameters used in rollout may not be up to date, which impairs RL convergence; and (2) the length variation of trajectories introduces severe data skewness, leading to workload imbalance and degraded system performance. Existing systems fail to address these two concerns in a unified manner. Techniques that tightly control data staleness often constrain effective data skewness mitigation, while aggressive data skewness mitigation tends to exacerbate data staleness. As a result, systems are forced to trade off convergence for performance, or vice versa. To address this, we propose StaleFlow, an RL post-training system that jointly tackles data staleness and skewness. First, to control staleness, StaleFlow introduces a global consistency protocol that tracks the full lifecycle of each trajectory and constrains staleness. Second, to mitigate skewness, StaleFlow re-designs the RL system architecture by constructing data servers for trajectories and parameters to achieve flexible rollout coordination. Subsequently, we develop a suite of staleness-aware, throughput-oriented strategies to enhance system performance. Evaluations show that StaleFlow achieves up to 1.42-2.68$\times$ (1.17-2.01$\times$ on average) higher throughput than state-of-the-art systems, without compromising convergence.

Figures (25)

• Figure 1: (Left) RL workflow in a fully disaggregated architecture. (Right) Time breakdown of post-training a Qwen3-30B-A3B (64 H20 GPUs). Rollout and training overlap and jointly dominate the time.
• Figure 2: Comparison of different RL systems. Higher values on the vertical axis indicate stronger support for rollout coordination.
• Figure 3: Effect of staleness bounds ($\eta$) on post-training Qwen2.5-MATH-7B with DAPO dapo (64 H20 GPUs). A larger staleness bound reduces idle waiting in training but slows convergence.
• Figure 4: A rollout step of Qwen3-30B-A3B (128 H20 GPUs). (Left) Trajectory and token distributions are highly skewed both within each instance and across different instances. (Right) This causes intra-instance underutilization and inter-instance idle waiting.
• Figure 5: Rollout coordination techniques for mitigating skewness.