POPE: Learning to Reason on Hard Problems via Privileged On-Policy Exploration

TL;DR

This work tackles the difficulty of training LLMs on hard reasoning problems with on-policy RL, where standard exploration strategies fail to yield reward. It introduces Privileged On-Policy Exploration (POPE), which uses short prefixes of oracle solutions to guide on-policy rollouts without using the oracle as a training target, enabling transfer from guided to unguided hard problems. Through a theoretical mental model and extensive experiments, POPE is shown to mitigate ray interference, expand the set of solvable problems, and improve performance on challenging benchmarks (AIME 2025, HMMT 2025) and other hard problem sets. The approach also demonstrates that transferring from guided to unguided problems relies on an overlap between guided and unguided states facilitated by instruction-following and backtracking, offering a practical and scalable path to stronger reasoning in LLMs.

Abstract

Reinforcement learning (RL) has improved the reasoning abilities of large language models (LLMs), yet state-of-the-art methods still fail to learn on many training problems. On hard problems, on-policy RL rarely explores even a single correct rollout, yielding zero reward and no learning signal for driving improvement. We find that natural solutions to remedy this exploration problem from classical RL, such as entropy bonuses, more permissive clipping of the importance ratio, or direct optimization of pass@k objectives, do not resolve this issue and often destabilize optimization without improving solvability. A natural alternative is to leverage transfer from easier problems. However, we show that mixing easy and hard problems during RL training is counterproductive due to ray interference, where optimization focuses on already-solvable problems in a way that actively inhibits progress on harder ones. To address this challenge, we introduce Privileged On-Policy Exploration (POPE), an approach that leverages human- or other oracle solutions as privileged information to guide exploration on hard problems, unlike methods that use oracle solutions as training targets (e.g., off-policy RL methods or warmstarting from SFT). POPE augments hard problems with prefixes of oracle solutions, enabling RL to obtain non-zero rewards during guided rollouts. Crucially, the resulting behaviors transfer back to the original, unguided problems through a synergy between instruction-following and reasoning. Empirically, POPE expands the set of solvable problems and substantially improves performance on challenging reasoning benchmarks.

Figures (11)

• Figure 1: A schematic of our approach, Privileged On-Policy Exploration (POPE) compared with other approaches for training on hard problems, where standard on-policy RL largely fails to produce successful rollouts. POPE uses oracle (e.g., human-written) solutions to solely guide on-policy exploration during RL, without ever training on the oracle information as targets (for e.g., via supervised fine-tuning or mid-training). We show that modifying standard RL training objectives to incentivize token-level exploration frequently introduce optimization pathologies. Our training approach (POPE) sidesteps such pathologies, while enabling learning from oracle information by "instructing" the model to build upon it.
• Figure 2: Interference schaul2019ray. In on-policy RL, training on a mixture of easy and hard problems preferentially accelerates progress on easy problems, often stalling or degrading performance on hard ones. This imbalance leads to plateaus during training; an ideal approach would induce more "uniform" progress across all problems.
• Figure 3: Left: Evolution of the fraction of solvable problems (measured via the pass@8 at 16k output length). Right: average token-level entropy statistics over the course of RL training. Observe that all of these representative classical exploration methods make similar amounts of (few) problems solvable, while creating pathologies in optimization in the sense that entropy blows up. We do notice large sensitivity to the clip threshold $\epsilon_{\text{high}}$ in our runs.
• Figure 4: No meaningful transfer from learning easy problems to hard problems.(a) evolution of the fraction of solvable problems (measured via pass@8 at 16k response length). (b) average training reward on easy problems mixed in training. (c) average token-level entropy over the course of RL training. Since we do not use an entropy bonus, entropy generally remains stable (or slightly decreases) throughout training. Observe that the fraction of solvable problems increases the most when using our guidance-based approach, "hard + guide". In contrast, incorporating easy prompts does not improve solvability of hard problems, providing a negative result for the transfer hypothesis for improving exploration on hard problems.
• Figure 5: Didactic two-problem experiment illustrating ray interference. We train on a setting consisting of one easy and one hard problem. (a) Success rate on the easy problem versus training steps. All methods rapidly solve the easy problem. (b) Optimization trajectories visualized by plotting $J(\pi_\theta;\text{easy})$ and $J(\pi_\theta;\text{hard})$ jointly over training. Mixing in an unrelated easy problem leads to rapid improvement on the easy problem at the cost of stagnation on the hard problem, illustrating negative transfer due to ray interference. (c) Using a related easy problem partially mitigates this effect, but remains inefficient and requires many more training steps to solve the hard problem compared to training on the hard problem alone. Our approach ("hard + guide") is the only one that improves convergence speed on the hard problem of all methods. (d) Success rate on the hard problem vs. the number of rollouts allocated to it. Beyond interference, POPE improves sample efficiency by reducing the number of rollouts required to learn the hard problem, indicating an acceleration in solvability of the hard problem.

Theorems & Definitions (1)

• Definition 2.1: Hard and easy problems