Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models

TL;DR

The paper addresses residual prompts in reinforcement learning with verifiable rewards (RLVR) for reasoning LLMs, where longer training and larger models leave many prompts with zero training signal. It introduces Explore Residual Prompts in Policy Optimization (ERPO), which combines Reactivate Training Signal (RA) and per-prompt temperature adaptation guided by a history tracker to reactivate and explore residual prompts. Across math-reasoning benchmarks, ERPO consistently outperforms GRPO-based baselines and RA, with especially strong gains on data less affected by contamination (e.g., AIME2025) and scalable improvements on larger models. This work demonstrates that carefully reusing residual prompts through adaptive exploration can enhance training diversity and reasoning capabilities in RLVR settings.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach for improving the reasoning abilities of large language models (LLMs). The Group Relative Policy Optimization (GRPO) family has demonstrated strong performance in training LLMs with RLVR. However, as models train longer and scale larger, more training prompts become residual prompts, those with zero variance rewards that provide no training signal. Consequently, fewer prompts contribute to training, reducing diversity and hindering effectiveness. To fully exploit these residual prompts, we propose the Explore Residual Prompts in Policy Optimization (ERPO) framework, which encourages exploration on residual prompts and reactivates their training signals. ERPO maintains a history tracker for each prompt and adaptively increases the sampling temperature for residual prompts that previously produced all correct responses. This encourages the model to generate more diverse reasoning traces, introducing incorrect responses that revive training signals. Empirical results on the Qwen2.5 series demonstrate that ERPO consistently surpasses strong baselines across multiple mathematical reasoning benchmarks.

Figures (4)

• Figure 1: Comparison between ERPO and GRPO. During RL training, the policy gradually learns from the training data, resulting in more residual prompts with all-correct responses. In GRPO, residual prompts yield zero-variance rewards and thus provide no training signal for policy updates, reducing the effectiveness of training data. In contrast, ERPO maintains a tracker for each prompt to record the number of times it produces all-correct responses, and adaptively encourages exploration on residual prompts to trigger incorrect responses and reactivate the training signal.
• Figure 2: The average and maximum sampling temperatures during the ERPO training process. The steps shown here are the prompt generation steps.
• Figure 3: The number of residual prompts with all-correct responses and prompts with all-incorrect responses during the ERPO training process. The steps shown here are the prompt generation steps.
• Figure 4: Performance of $mean@32$ on AIME2025.