A Relative-Budget Theory for Reinforcement Learning with Verifiable Rewards in Large Language Model Reasoning

TL;DR

The paper introduces a relative-budget framework for reinforcement learning with verifiable rewards (RLVR) that expresses compute budget as $ξ = H / \mathbb{E}[T]$, linking task difficulty to sample efficiency. It derives regime-dependent anti-concentration bounds and analyzes a gamma-model for time-to-solution, identifying a phase transition around $ξ \approx 1$ and an optimal budget in $[1.5,2.0]$ for learning efficiency and reasoning performance. The work provides finite-sample guarantees for online RL, outlines how relative budget governs monotonic improvement, and shows that online RL can drive linear growth of budget while maintaining learning progress, albeit with increasing per-iteration sample costs in the ample regime. Experiments across multiple LLMs and benchmarks validate the theory, revealing robust phase-transition behavior and guiding practical compute allocation for RLVR.

Abstract

Reinforcement learning (RL) is a dominant paradigm for improving the reasoning abilities of large language models, yet its effectiveness varies across tasks and compute budgets. We propose a \emph{relative-budget} theory explaining this variation through a single quantity called relative budget $ξ:= H/\mathbb{E}[T]$, where $H$ is the generation horizon (token budget) and $T$ denotes the number of tokens until the first correct solution under a base policy. We show that $ξ$ determines sample efficiency by controlling reward variance and the likelihood of informative trajectories. Our analysis reveals three regimes: in the \emph{deficient} regime ($ξ\to 0$), informative trajectories are rare and the sample complexity explodes; in the \emph{balanced} regime ($ξ=Θ(1)$), informative trajectories occur with non-negligible probability and RL is maximally sample-efficient; and in the \emph{ample} regime ($ξ\to \infty$), learning remains stable but marginal gains per iteration diminish. We further provide finite-sample guarantees for online RL that characterize learning progress across these regimes. Specifically, in a case study under idealized distributional assumptions, we show that the relative budget grows linearly over iterations. Our empirical results confirm these predictions in realistic settings, identifying a budget $ξ\in [1.5, 2.0]$ that maximizes learning efficiency and coincides with peak reasoning performance.

Figures (12)

• Figure 1: Conceptual illustration of our relative budget theory. The balanced relative budget regime ($\xi \approx 1$) acts as a phase transition point. In this regime, RL achieves maximal sample efficiency, whereas SFT suffers from performance degradation.
• Figure 2: Relations between $\xi$ and $c_0(K, \xi; \varepsilon)$ with different $K$ and $\varepsilon$.
• Figure 3: Optimal relative budget $\xi^\star(K)$ depending on $K$.
• Figure 4: Reward variance and anti-concentration coefficient vs. relative budget $\xi$ for Llama-3.2-3B-Instruct on GSM8K. A phase transition occurs at $\xi \approx 1.0$, with the learning signal maximizing at $\xi \approx 1.5 \text{--} 1.8$ in our experiments.
• Figure 5: The relationship between relative budget $\xi$ and accuracy in LLMs trained via RLVR.

Theorems & Definitions (40)

• Definition 3.1: Anti-concentration
• Lemma 3.2: Sub-optimality gap of RL, Theorem 5.7 in setlurscaling
• Definition 4.1: Relative budget
• Lemma 5.1: Anti-concentration in terms of $T(\tau)$
• Theorem 5.2: Regime-dependent anti-concentration and an implication for RL
• Proposition 5.3: Statistics under Gamma model
• Remark 5.4: Asymptotics
• Theorem 6.2: One-step improvement by RL
• Theorem 6.3: Three regimes of relative budget in online RL
• Theorem 6.4: Linear relative budget growth under Gamma model