The Art of Scaling Reinforcement Learning Compute for LLMs

TL;DR

This work addresses the lack of principled methods to predict how RL compute scales for training large language models. It introduces ScaleRL, a best-practice RL recipe, and a principled sigmoidal compute–performance framework that enables extrapolation from modest runs to large-scale budgets. Through a massive empirical study (~400k GPU-hours) across design choices, ScaleRL demonstrates predictable scaling trajectories and surpasses prior recipes in both asymptotic performance and compute efficiency. The findings show that asymptotic ceilings are design-dependent while efficiency can be enhanced through robust, well-validated components, enabling reliable scaling of RL for reasoning tasks and beyond. The work provides a rigorous methodology and practical recipe for cost-effectively predicting and achieving scalable RL with LLMs, with broader implications for multi-task RL, MoE models, and long-context reasoning.

Abstract

Reinforcement learning (RL) has become central to training large language models (LLMs), yet the field lacks predictive scaling methodologies comparable to those established for pre-training. Despite rapidly rising compute budgets, there is no principled understanding of how to evaluate algorithmic improvements for scaling RL compute. We present the first large-scale systematic study, amounting to more than 400,000 GPU-hours, that defines a principled framework for analyzing and predicting RL scaling in LLMs. We fit sigmoidal compute-performance curves for RL training and ablate a wide range of common design choices to analyze their effects on asymptotic performance and compute efficiency. We observe: (1) Not all recipes yield similar asymptotic performance, (2) Details such as loss aggregation, normalization, curriculum, and off-policy algorithm primarily modulate compute efficiency without materially shifting the asymptote, and (3) Stable, scalable recipes follow predictable scaling trajectories, enabling extrapolation from smaller-scale runs. Combining these insights, we propose a best-practice recipe, ScaleRL, and demonstrate its effectiveness by successfully scaling and predicting validation performance on a single RL run scaled up to 100,000 GPU-hours. Our work provides both a scientific framework for analyzing scaling in RL and a practical recipe that brings RL training closer to the predictability long achieved in pre-training.

Figures (20)

• Figure 1: Predicatably Scaling RL compute to 100,000 GPU Hours (a) We run ScaleRL for $100\text{k}$ GPU hours on an 8B dense model, and $50\text{k}$ GPU hours on a 17Bx16 MoE (Scout). We fit a sigmoid curve (\ref{['eq:sigmoid_law']}) on pass rate ($\text{mean}@16$) on iid validation dataset up to $50\text{k}$ (and $16\text{k})$ GPU hours and extrapolate to $100\text{k}$ (and $45\text{k})$ on the 8B (Scout MoE) models respectively. We trained for $7400$ steps for 8B and $7100$ steps for Scout, which is $3.5\times$ larger than ProRL prorl. The extrapolated curve ($\times$ markers) closely follows extended training, demonstrating both stability at large compute and predictive fits--establishing ScaleRL as a reliable candidate for RL scaling. (b) Downstream evaluation on AIME-24 shows a consistent scaling trend for ScaleRL, thus generalizing beyond the training data distribution. Moreover, scaling model size substantially improves the downstream and asymptotic RL performance.
• Figure 2: ScaleRL is more scalable than prevalent RL methods. We fit sigmoid curves (Equation \ref{['eq:sigmoid_law']}) on iid validation dataset to commonly-used training recipes like DeepSeek (GRPO) guo2025deepseek, Qwen-2.5 (DAPO) yu2025dapo, Magistral rastogi2025magistral, and Minimax-M1 minimax_m1, and compare them with ScaleRL. ScaleRL surpasses all other methods, achieving an asymptotic reward of $A=0.61$. Stars denote evaluation points; solid curves show the fitted curve over the range used for fitting; dashed curves extrapolate beyond it. We validate the predictability by running each method for longer ("$\times$" markers), which align closely with the extrapolated curves for stable recipes like ScaleRL and MiniMax. Further description of the individual recipes compared are given in Appendix \ref{['appendix:prevelant_methods']}.
• Figure 3: Interpreting \ref{['eq:sigmoid_law']}. We provide an example fit illustrating the roles of parameters $A$, $B$, and $C_{\text{mid}}$. $C_{\text{mid}}$ determines the compute point at which half of the total gain is achieved - smaller values correspond to faster ascent toward the asymptote. $B$ controls the curve’s steepness, with larger values indicating greater efficiency. $A$ represents the asymptotic performance reached at large compute scales. Further discussion is provided in Appendix \ref{['appendix:interpreting_fit']}.
• Figure 4: (a) Comparing "compute-scaling" of asynchronous off-policy RL setups. We report only the $B$ (scaling exponent) and $A$ (asymptotic pass rate) parameters of the fitted sigmoid curve (Equation \ref{['eq:sigmoid_law']}). PipelineRL-$k$ is much more efficient and slightly better in the large compute limit. (b) Different max off-policyness with PipelineRL.
• Figure 5: (a) Comparing popular loss functions: DAPO yu2025dapo, GSPO gspo, and CISPO minimax_m1. We find CISPO/GSPO achieve a higher asymptotic reward compared to DAPO. (b) Using FP32 precision in the final layer (LM head) gives a considerable boost in the asymptotic reward.