AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUs

TL;DR

AReaL-Hex tackles the challenge of efficiently training RL for LLMs on heterogeneous GPUs by introducing a heterogeneity-aware, fully asynchronous scheduling framework. It (i) characterizes the intrinsic CPU/GPU and memory bottlenecks of rollout generation versus policy training, (ii) formulates a constrained optimization problem and solves it with a two-phase approach: MILP-based search for per-stage parallelism and a graph-partitioning repartitioning step to balance resources under data staleness bounds, and (iii) demonstrates substantial end-to-end throughput gains (up to 1.50×) and cost reductions (up to 1.46×) over homogeneous baselines across model sizes 1.5B–14B. The work leverages specialized matching of HBM IO-bound rollout to memory-bandwidth-efficient GPUs and compute-bound training to acceleration-capable GPUs, delivering practical improvements for real-world RL workloads. These results suggest heterogeneous GPU deployments, when paired with principled scheduling, can meaningfully reduce time-to-train and cost, broadening access to RL-enhanced LLM capabilities.

Abstract

Maximizing training throughput and cost-efficiency of RL for LLMs is essential to democratize this advanced technique. One promising but challenging approach is to deploy such a computational workflow over heterogeneous GPUs. Unlike conventional large-scale LLM pretraining, RL training generally decomposes into three coupled stages, i.e., rollout generation, reward computation, and policy/value updates, which exhibit markedly different compute intensities, memory footprints, and communication patterns. Recent research shows that fully asynchronous RL training can disaggregate these stages across disjoint hardware pools without sacrificing training stability, creating a great opportunity for real-world heterogeneous deployment. To this end, we present AReaL-Hex, a heterogeneity-aware asynchronous RL training system that effectively schedules how to execute rollout generation and policy model training over heterogeneous GPUs while enforcing data staleness bounds. Concretely, we use a two-phase scheduler: (i) a constrained search with MILP to select per-stage parallelization strategies and workload assignments given a resource budget, and (ii) a graph-partitioning step that allocates heterogeneous GPUs and interconnects to maximize end-to-end throughput. Built atop a fully asynchronous RL architecture, AReaL-Hex maps HBM-I/O-bound generation and compute-bound optimization to more cost-efficient resources and balances their producer-consumer interactions to avoid both idleness and stale rollout trajectories. On the mathematical reasoning task with various model scales (1.5B, 7B, and 14B), compared to homogeneous deployments of state-of-the-art asynchronous RL systems: (i) When maintaining the same total budgets, AReaL-Hex delivers up to 1.50x higher training throughput; (ii) When achieving the same training throughput, AReaL-Hex results in up to 1.46x reduction in training cost.

Figures (5)

• Figure 1: Left: Illustration of the asynchronous RL training workflow. The computations for rollout generation and model training are overlapped. When a new version of model weights becomes available, both rollout and training processes are temporarily paused to update the rollout workers’ weights. Right: An example of scheduling results. Multiple rollout replicas, potentially configured with distinct parallelization strategies and hardware resources, are created to perform rollout generation tasks. (i) Once rollouts are completed, they are transmitted to the training workers. (ii) When the updated model weights are ready, training workers broadcast the new weights to rollout workers via NCCL collective communication operations.
• Figure 2: Rollout inference (short as Inf) and model training (short as Train) execution latency comparison between homogeneous setting 1, 2 and heterogeneous setting across different model scales.
• Figure 3: We present the end-to-end experimental results of AReaL-Hex operating on H800 and H20 mixed heterogeneous GPUs compared to AReaL operating on homogeneous H800 or H20 GPUs. All experimental settings maintain the same total budgets using the same prices as in §\ref{['sec:case']}.
• Figure 4: We present a breakdown of experiments comparing AReaL-Hex running on a 56-GPU heterogeneous cluster against AReaL running on a 24-GPU H800 homogeneous cluster. The terms Infer and Train denote the execution latencies for rollout generation and model training.
• Figure 5: Case study of AReaL-Hex's performance cost-efficiency across different cluster sizes ranging from 24 to 56 GPUs. For the H20 and H800 GPU per-hour costs, we follow the prior practice zhu2025megascale.