Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards

TL;DR

This work tackles the exploration ceiling observed in Reinforcement Learning with Verifiable Rewards by introducing PSN-RLVR, which perturbs policy parameters before rollout to generate temporally coherent, trajectory-level exploration. It integrates Truncated Importance Sampling to stabilize off-policy updates and a lightweight, real-time adaptive noise scheduler to replace costly KL-based control, demonstrating strong gains on GRPO across math-oriented benchmarks under large sampling budgets. The approach enhances semantic and operation diversity, enabling PSN-GRPO to discover qualitatively new reasoning strategies and outperform action-space noise baselines while remaining orthogonal to other RLVR enhancements. Overall, PSN-RLVR advances long-horizon reasoning in RLVR by enabling broader exploration of the solution space with efficient adaptation and stable learning, offering practical gains for verifiable math tasks and potential applicability to other structured reasoning domains.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) improves LLM reasoning, yet growing evidence indicates an exploration ceiling: it often reweights existing solution traces rather than discovering new strategies, limiting gains under large sampling budgets (e.g., pass-at-256). We address this limitation with PSN-RLVR, which perturbs policy parameters before rollout generation to induce temporally consistent, trajectory-level exploration that better preserves long-horizon chain-of-thought coherence than action-space noise. To mitigate the resulting sampling-update mismatch, we incorporate truncated importance sampling (TIS). To avoid expensive KL-based adaptive noise control, we propose a computationally efficient real-time adaptive noise scheduler driven by a lightweight surrogate that combines semantic diversity with normalized self-certainty. Instantiated on GRPO, a widely used RLVR method, PSN-GRPO consistently expands the effective reasoning capability boundary across multiple mathematical reasoning benchmarks and model families, yielding higher pass-at-k under large sampling budgets and outperforming prior exploration-oriented RLVR methods (e.g., Pass-at-k-style training) while remaining orthogonal and thus composable for additional gains.

Figures (10)

• Figure 1: Comparison of reasoning capability boundaries across different RLVR paradigms. (1) Standard RLVR-trained models (GRPO-Train) exhibit a significant reduction in semantic diversity and operation diversity compared to the base model (Qwen2.5-Math-7B). (2) Our proposed method, PSN-GRPO, restores and enhances this diversity, achieving significantly higher semantic and operation variance compared to the GRPO baseline. (3) In terms of reasoning performance, PSN-GRPO is superior to other exploration-focused methods, such as Pass@k trainingChen2025PasskTF and RLVR-Decomposedzhu2025surprisingeffectivenessnegativereinforcement, consistently delivering higher pass@k metrics, particularly under large sampling budgets (e.g., $k=128, 256$).
• Figure 2: Overview of the PSN-RLVR framework compared to Standard-RLVR. The noise-perturbed model $\pi_{\tilde{\theta}}$ generates rollouts to induce temporally consistent, trajectory-level exploration. The resulting reward signals are used to update the clean policy $\pi_{\theta}$
• Figure 3: We compare the pass@$k$ performance (top), semantic diversity , and operation diversity of PSN-GRPO against the standard GRPO baseline on Qwen2.5-Math-7B. PSN-GRPO achieves superior performance at large sampling budgets ($k \ge 16$). This gain is strongly correlated with increased semantic and operational diversity in generated trajectories dang2025assessing.
• Figure 4: Noise injection location (best settings). Average pass@k for the best noise scale at each injection place (Whole Layers, lm_head, and MLP). MLP injection attains the largest gains at high $k$.
• Figure 5: Average performance across benchmarks with different training time temperature. Increasing temperature beyond $1.5$ degrades overall performance.