Deep Dense Exploration for LLM Reinforcement Learning via Pivot-Driven Resampling

TL;DR

DEEP-GRPO tackles the exploration bottleneck in online reinforcement learning for large language models by pivot-driven dense resampling: it identifies deep, recoverable pivot states within failed trajectories, guides pivot selection with an online utility $P_{\phi}$ that balances recoverability and depth via $ abla$-based estimation, and performs dense local resampling from chosen pivots. A dual-stream optimization decouples global policy learning from local corrective updates, using gradient masking to avoid prefix duplication and stabilize training. Empirical results on GSM8K and multiple mathematical benchmarks show that DEEP-GRPO consistently outperforms GRPO and tree-based methods, maintains higher policy entropy, and yields longer, self-correcting responses. The approach reduces wasteful root-rollouts and provides a principled mechanism to exploit hard, error-prone regions of the reasoning space with practical gains for LLM reasoning tasks.

Abstract

Effective exploration is a key challenge in reinforcement learning for large language models: discovering high-quality trajectories within a limited sampling budget from the vast natural language sequence space. Existing methods face notable limitations: GRPO samples exclusively from the root, saturating high-probability trajectories while leaving deep, error-prone states under-explored. Tree-based methods blindly disperse budgets across trivial or unrecoverable states, causing sampling dilution that fails to uncover rare correct suffixes and destabilizes local baselines. To address this, we propose Deep Dense Exploration (DDE), a strategy that focuses exploration on $\textit{pivots}$-deep, recoverable states within unsuccessful trajectories. We instantiate DDE with DEEP-GRPO, which introduces three key innovations: (1) a lightweight data-driven utility function that automatically balances recoverability and depth bias to identify pivot states; (2) local dense resampling at each pivot to increase the probability of discovering correct subsequent trajectories; and (3) a dual-stream optimization objective that decouples global policy learning from local corrective updates. Experiments on mathematical reasoning benchmarks demonstrate that our method consistently outperforms GRPO, tree-based methods, and other strong baselines.

Figures (11)

• Figure 1: Comparison of Exploration Strategies. (a) GRPO naively scales up rollouts from the root, wasting budget on redundant high-probability paths while failing to explore deep states. (b) Tree-based methods perform dispersed branching with sparse samples, leading to limited local exploration and unstable local advantage estimation. (c) DEEP-GRPO (Ours) identifies critical "pivot" states and concentrates dense resampling there. We decouple optimization into global and local streams (with gradient masking), combining global policy learning with local refinement.
• Figure 2: Diminishing Returns of Naive Scaling. We evaluate GRPO on GSM8K by scaling its number of rollouts per prompt ($N$) from 8 to 64. While performance improves from $N=8$ to $N=16$, it quickly saturates, with negligible gains observed for $N=32$ and $N=64$. This demonstrates the diminishing returns of simply increasing the root-level sampling budget, which inefficiently re-explores previously discovered, high-probability trajectories. In contrast, our proposed method, DEEP-GRPO (p1b8), strategically allocates budget by selecting a pivot on error trajectory and branching 8 new rollouts from that point. This targeted exploration probes deeper, hard-to-reach, error-prone states. The results demonstrate that DEEP-GRPO effectively converts the additional computational budget into a significant performance improvement.
• Figure 3: Visualization of Sampling Distributions. The sampling probability $\mathcal{Q}(t)$ under different configurations. Uniform ($\gamma=0,P_\phi \approx const$) selects from all candidate branching points with equal probability. Root-Centric ($\gamma \ll 0$) concentrates mass at the root ($t=0$), mimicking the behavior of simply scaling root rollouts. Deep Bias ($\gamma=1, 2$) shifts focus to later positions, prioritizing deep states.
• Figure 4: Method Overview.
• Figure 5: Training dynamics on GSM8K. DEEP-GRPO achieves significant improvements in test accuracy compared to GRPO. Notably, our method maintains higher entropy and produces longer responses throughout the training.