OrchestrRL: Dynamic Compute and Network Orchestration for Disaggregated RL

TL;DR

This work addresses throughput and cost challenges in disaggregated RL by co-designing compute and network orchestration. It introduces OrchestrRL, combining a two-level compute scheduler (proactive MILP planning with ARIMA-based workload prediction and reactive load balancing) and RFabric, a dynamic hybrid EPS-OCS network fabric that adapts topology to Train, Gen, and weight synchronization phases. Empirical results on a 48-GPU testbed show up to 1.40x end-to-end throughput gains, while large-scale simulations demonstrate RFabric achieving near non-blocking Fat-tree performance with 2.2x–3.1x better cost efficiency than static networks. The work delivers a practical, scalable approach to optimizing disaggregated RL workloads, with significant implications for data-center network provisioning and large-scale RL deployments.

Abstract

Post-training with reinforcement learning (RL) has greatly enhanced the capabilities of large language models. Disaggregating the generation and training stages in RL into a parallel, asynchronous pipeline offers the potential for flexible scaling and improved throughput. However, it still faces two critical challenges. First, the generation stage often becomes a bottleneck due to dynamic workload shifts and severe execution imbalances. Second, the decoupled stages result in diverse and dynamic network traffic patterns that overwhelm conventional network fabrics. This paper introduces OrchestrRL, an orchestration framework that dynamically manages compute and network rhythms in disaggregated RL. To improve generation efficiency, OrchestrRL employs an adaptive compute scheduler that dynamically adjusts parallelism to match workload characteristics within and across generation steps. This accelerates execution while continuously rebalancing requests to mitigate stragglers. To address the dynamic network demands inherent in disaggregated RL -- further intensified by parallelism switching -- we co-design RFabric, a reconfigurable hybrid optical-electrical fabric. RFabric leverages optical circuit switches at selected network tiers to reconfigure the topology in real time, enabling workload-aware circuits for (i) layer-wise collective communication during training iterations, (ii) generation under different parallelism configurations, and (iii) periodic inter-cluster weight synchronization. We evaluate OrchestrRL on a physical testbed with 48 H800 GPUs, demonstrating up to a 1.40x throughput improvement. Furthermore, we develop RLSim, a high-fidelity simulator, to evaluate RFabric at scale. Our results show that RFabric achieves superior performance-cost efficiency compared to static Fat-Tree networks, establishing it as a highly effective solution for large-scale RL workloads.

Figures (14)

• Figure 1: Overview of RL workflow in different paradigms with GRPO algorithm shao2024deepseekmathgrpo.
• Figure 2: Sequence-length effects on generation runtime and load imbalance.
• Figure 3: Generation dynamics shaped by request completion and length variability.
• Figure 4: Spatial network traffic across RL stages. For weight synchronization, we use a optimized two-stage scheme: the Train DP group transmits weights once to the DP-0 group of each Gen pod, after which each Gen DP-0 broadcasts locally to its DP peers.
• Figure 5: Distribution of slack durations across communication operation types.