MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learning

TL;DR

This work tackles instability in group-relative policy optimization for LLMs under small rollout budgets, where baseline noise can flip advantages and misguide updates. It introduces MC-GRPO, which uses a median-centered within-prompt baseline computed from $G{+}1$ rollouts and excludes the median (zero-advantage) completion to maintain an effective update size of $G$, thereby reducing sign flips and improving stability. Across GRPO, DAPO, and DR-GRPO, and over diverse model families and math-reasoning tasks, MC-GRPO yields faster training convergence and higher final accuracy in the low-rollout regime, while remaining competitive at larger budgets and improving out-of-distribution performance. The method is a lightweight, plug-in improvement with practical impact for resource-constrained RL in reasoning tasks, and is complemented by analyses underscoring the link between sign-flip mitigation and learning quality, along with ablations validating the role of median-centering versus extra sampling.

Abstract

Group-relative policy optimization methods train language models by generating multiple rollouts per prompt and normalizing rewards with a shared mean reward baseline. In resource-constrained settings where the rollout budget is small, accuracy often degrades. We find that noise in the shared baseline induces advantage sign flips, where some rollouts receive an incorrect advantage sign, and the update direction is reversed. To address this, we propose Median-Centered Group Relative Policy Optimization (MC-GRPO), a simple and effective solution for small-rollout training. Our main idea is to replace the mean baseline with a median baseline: the median is far less sensitive to outlier rewards than the mean, mitigating the sign flips under small rollout size (G). We generate one additional rollout for median reference (G+1), and compute advantages by using the group median. With an odd-sized group, exactly one completion is the median and receives zero advantage, we exclude this pivot rollout from backpropagation so the number of gradient-contributing samples per prompt remains G, preserving the core update cost of standard G-rollout training. Across various GRPO-family methods and a wide range of models and scales, this median-centered training consistently improves stability and final accuracy in the low-rollout regime, reducing the gap between G=2 and G=8 to within 1%. Code is available at https://github.com/lotusroot-kim/MC-GRPO

Figures (7)

• Figure 1: Accuracy (%) versus the number of rollouts for Qwen3-1.7B trained on GSM8K. We compare the original GRPO, DAPO, and DR-GRPO methods (; baselines) with their Median-Centered (MC) variants (; ours). MC training improves robustness and yields larger gains under small rollout budgets (2$\sim$4 rollouts), while remaining competitive at higher rollout counts.
• Figure 2: Sign flips are frequent under small rollout budgets. (a) With few rollouts, the sample-mean baseline can shift substantially depending on which rollouts are included, causing an advantage sign flip for the same trajectory (e.g., the $0.5$-reward sample flips sign when the rollout set changes from $8$ to $4$). We also report the sign-flip rate (i.e. the fraction of rollouts whose advantage sign under a $k$-rollout baseline disagrees with an oracle sign computed from $G_{\mathrm{ref}}{=}128$ rollouts) for (b) Qwen3-1.7B/GSM8K, (c) Qwen2.5-7B-Instruct/Math-500, and (d) Llama-3.2-3B-instruct/Math-500. For each setting, we evaluate 250 prompts; for each prompt and $k\in\{2,4,8\}$, we draw 20 random $k$-subsamples from the 128 rollouts, compute either the mean or median baseline, and average the resulting sign-flip rates.
• Figure 3: Injected sign flips causally degrade GRPO training. Qwen3-1.7B is trained on GSM8K under a fixed rollout budget ($G{=}8$), while we synthetically inject sign noise by flipping the sign of a fraction $\rho$ of within-group advantages during training. Increasing $\rho$ consistently reduces final GSM8K accuracy, showing that the advantage sign flips directly corrupt the update direction and harm optimization.
• Figure 4: MC-GRPO overview. Given a prompt $q$, the policy samples $G{+}1$ completions and obtains rewards via a reward model (and, when applicable, a reference-model term). We compute group advantages by median-centering around $b(q)=\mathrm{median}(r_1,\dots,r_{G+1})$, which provides a robust shared baseline and reduces sensitivity to occasional high-reward outliers under small rollout budgets. To keep the effective update size fixed at $G$ trajectories per prompt, we generate one additional completion to define a unique median and then remove the median (zero-advantage) completion from the gradient update. The resulting advantages are a drop-in replacement for the original group-normalized advantages in standard GRPO-family losses (GRPO/DAPO/DR.GRPO).
• Figure 5: MC-GRPO overview. Accuracy as a function of the rollout budget $G\in\{2,4,8\}$ for GRPO, DAPO, and DR-GRPO, and their median-centered variants (MC-GRPO, MC-DAPO, MC-DR-GRPO). Across all settings, median-centering yields the largest gains at small rollout budgets and remains competitive as $G$ increases.