Beyond Variance: Prompt-Efficient RLVR via Rare-Event Amplification and Bidirectional Pairing

TL;DR

This work addresses prompt selection for reinforcement learning with verifiable rewards (RLVR) under extreme data scarcity, focusing on mathematical reasoning tasks. It introduces a mechanism-level view where bidirectional tail-event signals guide learning, and shows that two prompts—one hard-but-solvable and one easy-but-brittle—can suffice when paired via Weighted GRPO (WGRPO). WGRPO applies group-normalized advantages to weighted binary outcomes, amplifying rare successes and rare failures to provide strong, informative updates while stabilizing optimization. The approach yields consistent improvements on Qwen2.5-Math-7B/Instruct across AIME 2025, AMC23, and MATH500, closely matching gains achieved with far larger prompt pools, and demonstrates the importance of signal structure over sheer prompt quantity for RLVR in low-data regimes.

Abstract

Reinforcement learning with verifiable rewards (RLVR) is effective for training large language models on deterministic outcome reasoning tasks. Prior work shows RLVR works with few prompts, but prompt selection is often based only on training-accuracy variance, leading to unstable optimization directions and weaker transfer. We revisit prompt selection from a mechanism-level view and argue that an effective minibatch should provide both (i) a reliable positive anchor and (ii) explicit negative learning signals from rare failures. Based on this principle, we propose \emph{positive--negative pairing}: at each update, we sample a hard-but-solvable $q^{+}$ and an easy-but-brittle prompt $q^{-}$(high success rate but not perfect), characterized by low and high empirical success rates under multiple rollouts. We further introduce Weighted GRPO, which reweights binary outcomes at the pair level and uses group-normalized advantages to amplify rare successes on $q^{+}$ into sharp positive guidance while turning rare failures on $q^{-}$ into strong negative penalties. This bidirectional signal provides informative learning feedback for both successes and failures, improving sample efficiency without suppressing exploration. On Qwen2.5-Math-7B, a single paired minibatch per update consistently outperforms a GRPO baseline that selects two prompts via commonly used variance-based selection heuristics: AIME~2025 Pass@8 improves from 16.8 to 22.2, and AMC23 Pass@64 from 94.0 to 97.0, while remaining competitive with large-scale RLVR trained from a pool of 1209 training prompts. Similar gains are observed on Qwen2.5-Math-7B-Instruct.

Figures (3)

• Figure 1: Overview of bidirectional prompt selection and WGRPO. A common low-data RLVR baseline is to prioritize "high-variance" prompts, which can be sensitive to sampling noise. We instead select a two-prompt positive--negative pair: a hard prompt where rare successes provide strong positive guidance, and an easy prompt where rare failures provide strong negative penalties. WGRPO contrastively amplifies these tail events across repeated rollouts, encouraging exploration while stabilizing update directions.
• Figure 2: Pass@$k$ curves on AIME 2025, AMC23, and MATH500 for Qwen2.5-Math-7B with Base Model, GRPO+DSR-sub, GRPO+$\{\pi_{1},\pi_{2}\}$, WGRPO+$\{\pi_{1209},p_{12}\}$. Except training on the large dataset (GRPO+DSR-sub), WGRPO+$\{\pi_{1209},p_{12}\}$ overall outperforms other methods across different $k$. On AMC23 at $k=32,64$ and MATH500 at $k=8,16,32,64$, WGRPO+$\{\pi_{1209},p_{12}\}$ shows the strongest performance.
• Figure 3: Pass@$k$ curves on AIME 2025, AMC23, and MATH500 for Qwen2.5-Math-7B-Instruct with Base Model, GRPO+DSR-sub, GRPO+$\{\pi_{1},\pi_{2}\}$, WGRPO+$\{\pi_{1209},p_{12}\}$. WGRPO+$\{\pi_{1209},p_{12}\}$ is comparable to other methods, but shows distinct gains on AIME 2025.

Theorems & Definitions (2)

• Proposition 1.1
• proof