Miner:Mining Intrinsic Mastery for Data-Efficient RL in Large Reasoning Models

TL;DR

This work tackles data inefficiency in critic-free RL for large reasoning models caused by positive homogeneous prompts yielding vanishing learning signals. It introduces Miner, which repurposes the policy’s intrinsic uncertainty into a safe, self-supervised reward and couples three innovations: (i) uncertainty-driven intrinsic rewards, (ii) token-level focal credit assignment, and (iii) adaptive advantage calibration. Across six math/medical reasoning benchmarks and two base architectures, Miner achieves state-of-the-art gains in Pass@1 and Pass@K without extra rollouts or reward models, demonstrating that latent uncertainty exploitation is both necessary and sufficient for scalable RLVR. The findings imply substantial practical impact for training data-efficient, reasoning-capable LLMs in diverse domains with limited supervision and computational overhead.

Abstract

Current critic-free RL methods for large reasoning models suffer from severe inefficiency when training on positive homogeneous prompts (where all rollouts are correct), resulting in waste of rollouts due to zero advantage estimates. We introduce a radically simple yet powerful solution to \uline{M}ine \uline{in}trinsic mast\uline{er}y (Miner), that repurposes the policy's intrinsic uncertainty as a self-supervised reward signal, with no external supervision, auxiliary models, or additional inference cost. Our method pioneers two key innovations: (1) a token-level focal credit assignment mechanism that dynamically amplifies gradients on critical uncertain tokens while suppressing overconfident ones, and (2) adaptive advantage calibration to seamlessly integrate intrinsic and verifiable rewards. Evaluated across six reasoning benchmarks on Qwen3-4B and Qwen3-8B base models, Miner achieves state-of-the-art performance among the other four algorithms, yielding up to \textbf{4.58} absolute gains in Pass@1 and \textbf{6.66} gains in Pass@K compared to GRPO. Comparison with other methods targeted at exploration enhancement further discloses the superiority of the two newly proposed innovations. This demonstrates that latent uncertainty exploitation is both necessary and sufficient for efficient and scalable RL training of reasoning models.

Figures (13)

• Figure 1: (a) Traditional GRPO algorithms produce a credible number of rollouts that do not contribute to RL updates, due to indistinguishable top rewards. (b) Miner introduce intrinsic rewards to each rollout, injecting beneficial dense reward signals, achieving the same peak performance with only 50% training steps, and up to 23% higher performance on Qwen3-4B-Base.
• Figure 2: Framework of Miner. We focus on introducing intrinsic rewards to positive homogeneous prompts (PH). Upper Center: We use sequence-level uncertainty computed via the old policy $\pi_{\rm old}$ as the intrinsic rewards, to reinforce correct yet uncertain rollouts, without overfitting to already-mastered sequences; Upper Right: Then, we leverage token-level focal credit assignment to specifically rewarding critical tokens, again skipping self-confident tokens; Lower Right: Finally, to balance the learning signals from two groups, we calibrate the advantage score to a predefined threshold, significantly enhancing data efficiency without disturbing normal learning progress.
• Figure 3: Comparison with other exploration-enhanced algorithms on normalized Pass@1 and Pass@K scores.
• Figure 4: (a) Ablation study with three innovations (Intrinsic Reward (IR), Focal Weighting (FW) and Advantage Calibration (AC)) of Miner on the Qwen3-4B base model. (b) Performance dynamics given sufficient inference budgets. Apart from fluctuation within the error bar, Miner achieves non-negligible and sound improvements. (c) Parallel test-time scaling comparison with other algorithms, where Miner consistently outperforms other baselines with over 5 absolute points. Shaded areas denote $\pm 1$ standard deviation over 10 runs.
• Figure 5: Pass@K scaling of Miner and Base model on Qwen3-4B (Upper) and Qwen3-8B (Lower) models, where Miner still demonstrates improvements for a sufficiently large $K$.