It Takes Two: Your GRPO Is Secretly DPO
Yihong Wu, Liheng Ma, Lei Ding, Muzhi Li, Xinyu Wang, Kejia Chen, Zhan Su, Zhanguang Zhang, Chenyang Huang, Yingxue Zhang, Mark Coates, Jian-Yun Nie
TL;DR
The paper reframes Group Relative Policy Optimization (GRPO) as a contrastive objective and uncovers a link to Direct Preference Optimization (DPO). It then proposes 2-GRPO, a two-rollout per prompt variant, showing that it preserves unbiased gradient estimates and the quality of standard GRPO while delivering major efficiency gains. Theoretical analyses detail how 2-GRPO implicitly normalizes advantages and how gradient variance can be managed by increasing the number of prompts in a batch. Empirically, 2-GRPO matches or closely approaches GRPO performance across math-reasoning benchmarks while reducing wall-clock time by at least 70% and reducing rollouts by roughly an order of magnitude, signaling a practical path toward resource-efficient RL for LLM post-training.
Abstract
Group Relative Policy Optimization (GRPO) is a prominent reinforcement learning algorithm for post-training Large Language Models (LLMs). It is commonly believed that GRPO necessitates a large group size to ensure stable training via precise statistical estimation, which incurs substantial computational overhead. In this work, we challenge this assumption by reframing GRPO as a form of contrastive learning, which reveals a fundamental connection to Direct Preference Optimization (DPO). Motivated by DPO's empirical success, we investigate the minimal two-rollout case (2-GRPO), a configuration previously deemed infeasible. We provide a rigorous theoretical analysis to validate 2-GRPO and demonstrate empirically that it achieves performance on par with 16-GRPO, despite using only 1/8 of the rollouts and reducing training time by over 70%.