Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration

TL;DR

This work identifies two under-explored dimensions in RLVR—Depth (the hardest solvable problems) and Breadth (instances per iteration)—and reveals a bias in GRPO where cumulative advantage underweights high-difficulty samples, capping Pass@K. It introduces Difficulty Adaptive Rollout Sampling (DARS) to reallocate compute toward hard problems and demonstrates that large-breadth training further boosts Pass@1 by sustaining token entropy; combining both yields superior performance across Pass@K and Pass@1 (DARS-Breadth). The results show depth and breadth as orthogonal and complementary dimensions, enabling substantial gains in LLM reasoning without prohibitive inference costs. Collectively, the approach advances RLVR toward deeper, more robust reasoning and points to practical strategies for scalable, self-improving LLMs.

Abstract

Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful paradigm for unlocking reasoning capabilities in large language models, yet its full potential is hindered by two under-explored dimensions: Depth-the hardest problem a model can sample; Breadth-the number of instances consumed in a single iteration. We dissect the popular GRPO algorithm and reveal a systematic bias: the cumulative-advantage disproportionately weights samples with medium accuracy, while down-weighting the low-accuracy instances that are crucial for pushing reasoning boundaries. To rectify the depth neglect, we introduce Difficulty Adaptive Rollout Sampling (DARS), which re-weights hard problems through targeted multi-stage rollouts, thereby increasing the number of positive rollouts for hard problems. Empirically, naively enlarging rollout size only accelerates convergence and even hurts Pass@K. Our DARS, in contrast, delivers consistent Pass@K gains without extra inference cost at convergence. Just as we adaptively expanded the depth of exploration, we now ask whether aggressively scaling the breadth of training data can further amplify reasoning gains. To this end, we intensely scale batch size and replace PPO's mini-batch iterations with full-batch updates over multiple epochs. Increasing breadth significantly enhances Pass@1 performance. Large-breadth training sustains high token-level entropy, indicating continued exploration and reduced gradient noise. We further present DARS-B, which augments DARS with large breadth, and demonstrate simultaneous gains in Pass@K and Pass@1. The results confirm that breadth and adaptive exploration across depth operate as orthogonal dimensions in RLVR, which are key to unleashing the reasoning power of RLVR.

Figures (14)

• Figure 1: Training dynamics of Pass@1 and Pass@K performance. We show that our DARS significantly improves Pass@K performance and is complementary to breadth scaling to further improve Pass@1 performance.
• Figure 2: Training dynamics of Pass@1 and Pass@K performance of Qwen2.5-Math-1.5b and Qwen2.5-Math-7b with different rollout size.
• Figure 3: Statistical results of cumulative advantage. Group relative advantage calculation methods underestimate high-difficulty problems. $n$ denotes group size.
• Figure 4: Training dynamics of Pass@1 and Pass@K performance of Qwen2.5-Math-1.5b and Qwen2.5-Math-7b with different batch size.
• Figure 5: Training dynamics of Pass@1 performance and token entropy for Qwen2.5-Math-1.5b and Qwen2.5-Math-7b.