EBPO: Empirical Bayes Shrinkage for Stabilizing Group-Relative Policy Optimization

TL;DR

GRPO suffers high estimator variance at small group sizes and vanishing gradients in saturated failure regimes when optimizing reasoning tasks with verifiable rewards. EBPO introduces Empirical Bayes shrinkage by blending the local group baseline with a globally updated prior $\mu_{glob}$ (via online Welford updates), yielding strictly lower MSE, non-vanishing penalties, and bounded entropy decay. Empirically, EBPO outperforms GRPO and baselines across AIME and OlympiadBench, with pronounced stability and sample efficiency at small $G$, especially when paired with difficulty-based curriculum via clustered sampling. This work delivers a robust, scalable framework for stabilizing online policy optimization in RLVR, enabling reliable gains in elite mathematical reasoning with limited compute.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for enhancing the reasoning capabilities of Large Language Models (LLMs). However, dominant approaches like Group Relative Policy Optimization (GRPO) face critical stability challenges: they suffer from high estimator variance under computational constraints (small group sizes) and vanishing gradient signals in saturated failure regimes where all responses yield identical zero rewards. To address this, we propose Empirical Bayes Policy Optimization (EBPO), a novel framework that regularizes local group-based baselines by borrowing strength from the policy's accumulated global statistics. Instead of estimating baselines in isolation, EBPO employs a shrinkage estimator that dynamically balances local group statistics with a global prior updated via Welford's online algorithm. Theoretically, we demonstrate that EBPO guarantees strictly lower Mean Squared Error (MSE), bounded entropy decay, and non-vanishing penalty signals in failure scenarios compared to GRPO. Empirically, EBPO consistently outperforms GRPO and other established baselines across diverse benchmarks, including AIME and OlympiadBench. Notably, EBPO exhibits superior training stability, achieving high-performance gains even with small group sizes, and benefits significantly from difficulty-stratified curriculum learning.

Figures (6)

• Figure 1: Overview of the EBPO Framework. Unlike standard GRPO which relies solely on the local group mean, EBPO computes a "Smart Baseline" (Shrinkage Estimator) by blending the noisy local group mean with a stable global prior (updated via global history). This allows for informative advantage estimates even in small groups or saturated failure regimes.
• Figure 2: Advantage Signal in Failure Scenarios ($r_i=0$ for all $i$). Comparison of the advantage signal of GRPO (dashed line) versus EBPO (solid blue line) when the model fails all attempts for a given prompt. While GRPO yields a vanishing gradient ($\hat{A}=0$) regardless of task difficulty, EBPO provides a dynamic penalty signal $-\mathcal{S}\mu_{\text{glob}}$ that scales with the global success rate ($\mu_{\text{glob}}$).
• Figure 3: Optimization stability analysis. (a) EBPO maintains healthy, non-vanishing gradients compared to GRPO's diminishing signals. (b) EBPO prevents policy collapse by strictly bounding the update magnitude (per-step KL) throughout training.
• Figure 4: Evolution of Policy Entropy ($G=4$). EBPO maintains a consistently higher policy entropy than GRPO, demonstrating that the global prior effectively sustains exploration throughout training.
• Figure 5: Validation Performance. While GRPO plateaus or degrades in late-stage training—indicative of overfitting or policy collapse—EBPO exhibits sustained performance gains.

Theorems & Definitions (13)

• Remark 3.1: Online Stability vs. Bias for Between Group Variances
• Theorem 3.2: Non-Vanishing Gradients in Saturation Regimes
• Theorem 3.3: Stability via MSE Reduction
• Remark 3.4
• Corollary 3.5: Global Context Sensitivity
• Proposition 3.6: Entropy Conservation via Shrinkage-Regularized Covariance
• Remark 3.7
• Proposition 3.8: Advantage of Clustered Sampling
• proof : Proof of Theorem \ref{['thm:non_vanishing']}
• proof : Proof of Theorem \ref{['thm:mse_stability']}