Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GVPO: Group Variance Policy Optimization for Large Language Model Post-Training

Kaichen Zhang, Yuzhong Hong, Junwei Bao, Hongfei Jiang, Yang Song, Dingqian Hong, Hui Xiong

TL;DR

GVPO tackles instability in KL-constrained LLM post-training by introducing a group-variance gradient weighting that cancels the partition function, ensuring alignment with the KL-constrained optimum. The method provides a rigorous theoretical guarantee of a unique global optimum equal to the KL-constrained reward maximizer and accommodates flexible off-policy sampling. Empirically, GVPO outperforms prior methods such as GRPO and DPO on math-reasoning benchmarks and a summarization task, with ablations demonstrating the critical roles of the variance and covariance regularizers. This work offers a principled, robust alternative for post-training that combines theoretical rigor with practical adaptability for large language models.

Abstract

Post-training plays a crucial role in refining and aligning large language models to meet specific tasks and human preferences. While recent advancements in post-training techniques, such as Group Relative Policy Optimization (GRPO), leverage increased sampling with relative reward scoring to achieve superior performance, these methods often suffer from training instability that limits their practical adoption. As a next step, we present Group Variance Policy Optimization (GVPO). GVPO incorporates the analytical solution to KL-constrained reward maximization directly into its gradient weights, ensuring alignment with the optimal policy. The method provides intuitive physical interpretations: its gradient mirrors the mean squared error between the central distance of implicit rewards and that of actual rewards. GVPO offers two key advantages: (1) it guarantees a unique optimal solution, exactly the KL-constrained reward maximization objective, (2) it supports flexible sampling distributions that avoids on-policy and importance sampling limitations. By unifying theoretical guarantees with practical adaptability, GVPO establishes a new paradigm for reliable and versatile LLM post-training.

GVPO: Group Variance Policy Optimization for Large Language Model Post-Training

TL;DR

GVPO tackles instability in KL-constrained LLM post-training by introducing a group-variance gradient weighting that cancels the partition function, ensuring alignment with the KL-constrained optimum. The method provides a rigorous theoretical guarantee of a unique global optimum equal to the KL-constrained reward maximizer and accommodates flexible off-policy sampling. Empirically, GVPO outperforms prior methods such as GRPO and DPO on math-reasoning benchmarks and a summarization task, with ablations demonstrating the critical roles of the variance and covariance regularizers. This work offers a principled, robust alternative for post-training that combines theoretical rigor with practical adaptability for large language models.

Abstract

Post-training plays a crucial role in refining and aligning large language models to meet specific tasks and human preferences. While recent advancements in post-training techniques, such as Group Relative Policy Optimization (GRPO), leverage increased sampling with relative reward scoring to achieve superior performance, these methods often suffer from training instability that limits their practical adoption. As a next step, we present Group Variance Policy Optimization (GVPO). GVPO incorporates the analytical solution to KL-constrained reward maximization directly into its gradient weights, ensuring alignment with the optimal policy. The method provides intuitive physical interpretations: its gradient mirrors the mean squared error between the central distance of implicit rewards and that of actual rewards. GVPO offers two key advantages: (1) it guarantees a unique optimal solution, exactly the KL-constrained reward maximization objective, (2) it supports flexible sampling distributions that avoids on-policy and importance sampling limitations. By unifying theoretical guarantees with practical adaptability, GVPO establishes a new paradigm for reliable and versatile LLM post-training.
Paper Structure (20 sections, 2 theorems, 23 equations, 4 figures, 5 tables, 1 algorithm)

This paper contains 20 sections, 2 theorems, 23 equations, 4 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Towards better computation leverage in post-training
  4. Optimal solution to the KL-constrained reward maximization
  5. Unified framework of post-training
  6. Group Variance Policy Optimization
  7. Motivation
  8. Method
  9. Theoretical guarantee
  10. Discussions with DPO
  11. Discussions with GRPO and Policy Gradient methods
  12. Experiments
  13. Related Work
  14. Conclusion
  15. Supplementary Experiment Information
  16. ...and 5 more sections

Key Result

Theorem 3.1

The unique optimal policy that minimizes $\hat{\mathcal{L}}_{\text{GVPO}}(\theta)$, defined as , is given by $\pi_\theta (y|x)=\pi^* (y|x)= \frac{1}{Z(x)}\pi_{\theta^\prime}(y|x)e^{R(x,y)/\beta}$ for $\pi_s=\pi_{\theta^\prime}$.

Figures (4)

  • Figure 1: Three equivalent loss functions of GVPO offer distinct interpretations: (1) The Negative log-Likelihood perspective (top) illustrates that GVPO accommodates broader sampling distributions compared to conventional policy gradient methods; (2) The Mean Squared Error interpretation (middle) reveals GVPO’s unique optimal solution, which simultaneously maximizes reward under a KL constraint; and (3) The Reinforcement Learning viewpoint (bottom) highlights GVPO’s implicit regularization terms that ensure stable policy optimization. We assume $\beta=1$ for simplicity.
  • Figure 2: Ablation on $\beta$. Each line represents a dataset.
  • Figure 3: Ablation on $k$. Blue line: GVPO; Red line: GRPO.
  • Figure 4: Ablation on $\pi_s$. #(historical $y$) : #($y$ from $\pi_{\theta_{\text{old}}}$)

Theorems & Definitions (3)

  • Theorem 3.1
  • Theorem 3.2
  • proof