Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purification

TL;DR

This paper tackles RLVR inefficiency by identifying interference tokens in prompts as a principal source of poor exploration under limited rollouts. It introduces Lens, a two-stage framework that purifies interference tokens and then calibrates policy optimization on the original noisy prompts using successful rollouts from purified prompts (CRPO) with sample reweighting. Empirical results across seven math benchmarks demonstrate that Lens outperforms GRPO and prompt-filtering baselines, delivering a 3.88% average improvement and up to 1.6x faster convergence with competitive or lower compute. The work highlights the importance of token-level prompt purification for robust, efficient reasoning under noisy real-world prompts and offers a new direction for RLVR research.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has advanced LLM reasoning, but remains constrained by inefficient exploration under limited rollout budgets, leading to low sampling success and unstable training in complex tasks. We find that many exploration failures arise not from problem difficulty, but from a small number of prompt tokens that introduce interference. Building on this insight, we propose the Less Noise Sampling Framework (LENS), which first prompts by identifying and removing interference tokens. then transfers successful rollouts from the purification process to supervise policy optimization on the original noisy prompts, enabling the model to learn to ignore interference in the real-world, noisy prompting settings. Experimental results show that LENS significantly outperforms GRPO, delivering higher performance and faster convergence, with a 3.88% average gain and over 1.6$\times$ speedup. Our work highlights the critical role of pruning interference tokens in improving rollout efficiency, offering a new perspective for RLVR research.

Figures (8)

• Figure 1: An example of interference token purification: Removing a few interference tokens corrects the reasoning rollout and turns it into a successful one.
• Figure 2: (a) Zero-Reward Prompt Analysis: Comparison of the zero-reward prompt ratio across different models and rollout sizes ($n$). Lens significantly reduces the proportion of zero-reward samples compared to GRPO, enhancing training efficiency. (b) Distribution of token-level Interference Scores (log scale): Only a few tokens exhibit high interference. (c) Rollout Accuracy Improvement: Removing these interference tokens leads to a significant improvement in rollout success rates (Average@8).
• Figure 3: Method Overview. In the first stage, Lens identifies and purifies interference tokens within low-success prompts via Interference Score (Defined in Section \ref{['sec:identification']}), thereby generating a higher proportion of successful rollouts. In the second stage, Lens uses successful rollouts from the denoised prompts as high-reward supervision to calibrate policy optimization on the original prompt, correcting gradient updates distorted by interference.
• Figure 4: Learning curves of Lens and GRPO across model scales and task difficulties. We compare Qwen3-4B/8B-Base backbones on MATH-500 (Medium) and OlympiadBench (High). Lens converges faster and achieves comparable or higher final accuracy than GRPO under the same training step, indicating more efficient optimization.
• Figure 5: Sampling accuracy distribution across three training phases. We compare the sampling distributions of GRPO, GRPO$_\text{extended}$ and Lens across the early, middle and late training stages.