RollArt: Scaling Agentic RL Training via Disaggregated Infrastructure

TL;DR

RollArt tackles the inefficiency of multi-task agentic RL training on heterogeneous, disaggregated hardware by introducing hardware-affinity workload mapping, trajectory-level asynchronous execution, and statefulness-aware computation. It presents a declarative programming model and a heterogeneity-aware runtime that route stages to best-fit hardware, overlap rollout and training at the trajectory level, and offload stateless reward computation to serverless platforms. Macro benchmarks show up to $2.05$x end-to-end speedup and a production deployment on over $3{,}000$ GPUs demonstrating scalability, resilience, and sustained throughput. The work provides a practical path toward scalable, production-grade agentic RL training and a public codebase at https://github.com/alibaba/ROLL.

Abstract

Agentic Reinforcement Learning (RL) enables Large Language Models (LLMs) to perform autonomous decision-making and long-term planning. Unlike standard LLM post-training, agentic RL workloads are highly heterogeneous, combining compute-intensive prefill phases, bandwidth-bound decoding, and stateful, CPU-heavy environment simulations. We argue that efficient agentic RL training requires disaggregated infrastructure to leverage specialized, best-fit hardware. However, naive disaggregation introduces substantial synchronization overhead and resource underutilization due to the complex dependencies between stages. We present RollArc, a distributed system designed to maximize throughput for multi-task agentic RL on disaggregated infrastructure. RollArc is built on three core principles: (1) hardware-affinity workload mapping, which routes compute-bound and bandwidth-bound tasks to bestfit GPU devices, (2) fine-grained asynchrony, which manages execution at the trajectory level to mitigate resource bubbles, and (3) statefulness-aware computation, which offloads stateless components (e.g., reward models) to serverless infrastructure for elastic scaling. Our results demonstrate that RollArc effectively improves training throughput and achieves 1.35-2.05\(\times\) end-to-end training time reduction compared to monolithic and synchronous baselines. We also evaluate RollArc by training a hundreds-of-billions-parameter MoE model for Qoder product on an Alibaba cluster with more than 3,000 GPUs, further demonstrating RollArc scalability and robustness. The code is available at https://github.com/alibaba/ROLL.

Figures (15)

• Figure 1: Disaggregated infrastructure for agentic RL training.
• Figure 2: Synchronous vs. asynchronous training.
• Figure 3: Breakdown of a training step: successful runs (top, avg=365.7s) versus execution with environment failures (bottom, avg=513.3s).
• Figure 4: End-to-end rollout time (seconds) of different tasks on H20 and H800 GPUs across varying batch sizes.
• Figure 5: The analysis of environment interaction: (a) Cumulative distribution function of time (log-scaled) taken for environment initialization (env.reset) and environment step (env.step). (b) Illustration of how long-tail environments affect multi-turn rollouts under batched env interaction.