F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

TL;DR

This work analyzes reinforcement learning with verifiable rewards (RLVR) under group-relative policy optimization and binary rewards, showing that a finite group size $N$ can cause sharpening by missing rare-correct trajectories. It derives a closed-form tail-miss probability $\,\Pr(\mathcal{B}_\tau)\,$ and characterizes how probability mass redistributes within the correct set, revealing that unsampled-correct mass can shrink even as total correct mass grows. To mitigate this, the authors introduce F-GRPO, a lightweight focal-weighting scheme $g(x) = (1 - \widehat{\mu}_{\mathrm{pos}}(x))^\gamma$ that down-weights updates on high-success prompts and can be applied to GRPO, DAPO, and CISPO without additional compute. Empirically, F-GRPO yields consistent pass@256 gains on in-domain math and OOD benchmarks across multiple model families, while preserving or improving pass@1, demonstrating improved solution diversity and robustness without extra cost. Overall, the paper provides both a theoretical lens on RLVR sampling dynamics and a practical drop-in method to maintain diversity in group-relative policy optimization for LLM reasoning tasks.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is commonly based on group sampling to estimate advantages and stabilize policy updates. In practice, large group sizes are not feasible due to computational limits, which biases learning toward trajectories that are already likely. Smaller groups often miss rare-correct trajectories while still containing mixed rewards, concentrating probability on common solutions. We derive the probability that updates miss rare-correct modes as a function of group size, showing non-monotonic behavior, and characterize how updates redistribute mass within the correct set, revealing that unsampled-correct mass can shrink even as total correct mass grows. Motivated by this analysis, we propose a difficulty-aware advantage scaling coefficient, inspired by Focal loss, that down-weights updates on high-success prompts. The lightweight modification can be directly integrated into any group-relative RLVR algorithm such as GRPO, DAPO, and CISPO. On Qwen2.5-7B across in-domain and out-of-domain benchmarks, our method improves pass@256 from 64.1 $\rightarrow$ 70.3 (GRPO), 69.3 $\rightarrow$ 72.5 (DAPO), and 73.2 $\rightarrow$ 76.8 (CISPO), while preserving or improving pass@1, without increasing group size or computational cost.

Figures (5)

• Figure 1: (a) Probability that a training update is active (mixed rewards in batch) yet misses rare-correct solutions, as a function of group size $N$. This probability peaks at intermediate $N$: small groups rarely produce learning signal, large groups cover rare modes, but moderate groups combine active updates with poor coverage. (b,c) Empirical consequences on AIME 2025 (math) and IFEval (OOD): GRPO at $N{=}8$ improves pass@1 over $N{=}2$ but degrades pass@256, consistent with the sharpening regime. F-GRPO at $N{=}8$ recovers pass@256 while maintaining pass@1, using $4{\times}$ less compute than $N{=}32$.
• Figure 2: Tail-miss probability $\Pr(\mathcal{B}_\tau)$ from Lemma \ref{['lem:P_Btau']} versus group size $N$. Each panel fixes $\mu_{\mathrm{pos}} \in \{0.8, 0.5, 0.2\}$; curves vary $\rho = \tau/\mu_{\mathrm{pos}}$, the fraction of correct mass in the rare-correct region. Stars mark peaks. For all parameter combinations, $\Pr(\mathcal{B}_\tau)$ peaks at intermediate $N$: small $N$ yields low activity, large $N$ yields good coverage, but moderate $N$ combines active groups with poor coverage of rare modes. Smaller $\rho$ shifts the peak rightward and upward.
• Figure 3: Scaled advantage magnitude $g(x) \cdot |\widehat{A}^{\mathrm{GRPO}}|$ versus success probability $\mu_{\mathrm{pos}}(x)$ for binary rewards. Solid lines: correct rollouts; dashed lines: incorrect rollouts. Higher $\gamma$ suppresses updates on high-success prompts, shifting gradient contribution toward prompts where the policy succeeds less frequently.
• Figure 4: Categorical policy simulation following brorl setup. (a) Total correct mass $Q_{\mathrm{pos}}$ vs. training step. (b) Retained positive mass $\mathcal{M}_{\mathrm{ret}}$ vs. step. (c) Final metrics vs. group size $N$, with three regimes: I slow $Q_{\mathrm{pos}}$ growth, diversity preserved; II concentration zone (shaded), $Q_{\mathrm{pos}}$ grows but $\mathcal{M}_{\mathrm{ret}}$ collapses; III both metrics high. Solid: $\gamma{=}0$; dashed: $\gamma{=}1$. $N{=}131\text{k}$ maintains $\mathcal{M}_{\mathrm{ret}}{\approx}1$ throughout, consistent with $\Pr(\mathcal{B}_\tau) < 10^{-3}$ (Appendix \ref{['app:categorical_sim']}).
• Figure 5: Tail-miss probability $\Pr(\mathcal{B}_\tau)$ versus group size $N$ for $\mu_{\mathrm{pos}} = 0.64$ and $\tau = 6.3 \times 10^{-5}$ (corresponding to a non-anchor correct action in the simulation). The non-monotonic shape explains the concentration zone: intermediate $N$ maximizes the probability that a correct action is unsampled while the batch contains mixed rewards.

Theorems & Definitions (8)

• Lemma 3.1
• Proposition 3.2
• proof
• Lemma 2.1
• proof
• Corollary 2.2
• proof
• proof