AMIR-GRPO: Inducing Implicit Preference Signals into GRPO

TL;DR

AMIR-GRPO introduces an implicit Direct Preference Optimization (DPO)-style contrastive regularizer into GRPO by treating intra-group rewards $\{r_i\}_{i=1}^G$ for a query $q$ as an ordering and forming pairwise preferences $\mathcal{S}(q) = \{(i,j) \mid r_i > r_j + \delta_r\}$. This augments the GRPO objective $\mathcal{J}_{GRPO}$ with a contrastive term $\mathcal{J}_{pref}$, weighted by a dynamically adapted $\lambda_{reg}$, to encourage higher probability for preferred completions than rejected ones. Empirically, on a suite of math-reasoning benchmarks (GSM8K, AIME, AMC, OlympiadBench, Minerva, AQUA-RAT, LiveMathBench), AMIR-GRPO achieves higher Pass@k and broader solvable-space coverage than standard GRPO, while reducing length bias and sharpening the decision boundary between correct and incorrect reasoning. The approach requires no extra annotations and remains compatible with GSPO and other GRPO variants, offering a scalable path to leverage implicit preference signals for improved mathematical reasoning in LLM post-training.

Abstract

Reinforcement learning has become the primary paradigm for aligning large language models (LLMs) on complex reasoning tasks, with group relative policy optimization (GRPO) widely used in large-scale post-training. However, GRPO faces structural limitations in reasoning-heavy settings: sequence-level advantage normalization introduces systematic length bias, penalties for low-quality trajectories are diluted, and the scalar objective discards rich pairwise preference information embedded in within-group reward rankings. As a result, valuable supervision from costly rollouts remains underutilized. We propose AMIR-GRPO, which augments GRPO with an implicit DPO-style contrastive regularizer constructed directly from intra-group reward rankings, requiring no additional annotations. This mechanism amplifies suppression of low-reward trajectories, attenuates response-level length bias, and transforms each rollout group into a denser set of supervision constraints. Across multiple mathematical reasoning benchmarks, AMIR-GRPO consistently outperforms strong GRPO baselines, yields clearer separation between correct and incorrect reasoning chains, and delivers broader coverage gains beyond the subset of instances solved by standard GRPO.

Figures (8)

• Figure 1: End-to-end training pipeline of AMIR-GRPO.
• Figure 2: Impact of $\beta_\text{DPO}$ across datasets. Values indicate $\Delta\text{Pass@}k$ relative to the GRPO baseline.
• Figure 3: Comparison of reasoning-trajectory perplexity on an AMC23 question.
• Figure 4: Preference margin separation.
• Figure 5: Error distribution.