Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

$f$-GRPO and Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignment

Rajdeep Haldar, Lantao Mei, Guang Lin, Yue Xing, Qifan Song

TL;DR

This work develops a unified divergence-based viewpoint for general LLM alignment that spans both verifiable-reward reinforcement learning (RLVR) and preference alignment (PA). It introduces on-policy f-GRPO and hybrid f-HAL losses derived from $f$-divergences, with theoretical guarantees of alignment consistency and average reward improvement. Through extensive experiments on Math Reasoning (RLVR) and Safety Alignment (PA), the framework demonstrates improved performance, robustness, and flexibility over existing methods, while mitigating reward hacking via hybrid objective design. The approach provides a practical, principled foundation for integrating on-policy reinforcement signals and offline preference data in a single, transferable RL framework. Overall, divergence estimation serves as a unifying tool for general LLM alignment with strong theoretical and empirical support.</nobr>

Abstract

Recent research shows that Preference Alignment (PA) objectives act as divergence estimators between aligned (chosen) and unaligned (rejected) response distributions. In this work, we extend this divergence-based perspective to general alignment settings, such as reinforcement learning with verifiable rewards (RLVR), where only environmental rewards are available. Within this unified framework, we propose $f$-Group Relative Policy Optimization ($f$-GRPO), a class of on-policy reinforcement learning, and $f$-Hybrid Alignment Loss ($f$-HAL), a hybrid on/off policy objectives, for general LLM alignment based on variational representation of $f$-divergences. We provide theoretical guarantees that these classes of objectives improve the average reward after alignment. Empirically, we validate our framework on both RLVR (Math Reasoning) and PA tasks (Safety Alignment), demonstrating superior performance and flexibility compared to current methods.

$f$-GRPO and Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignment

TL;DR

This work develops a unified divergence-based viewpoint for general LLM alignment that spans both verifiable-reward reinforcement learning (RLVR) and preference alignment (PA). It introduces on-policy f-GRPO and hybrid f-HAL losses derived from -divergences, with theoretical guarantees of alignment consistency and average reward improvement. Through extensive experiments on Math Reasoning (RLVR) and Safety Alignment (PA), the framework demonstrates improved performance, robustness, and flexibility over existing methods, while mitigating reward hacking via hybrid objective design. The approach provides a practical, principled foundation for integrating on-policy reinforcement signals and offline preference data in a single, transferable RL framework. Overall, divergence estimation serves as a unifying tool for general LLM alignment with strong theoretical and empirical support.</nobr>

Abstract

Recent research shows that Preference Alignment (PA) objectives act as divergence estimators between aligned (chosen) and unaligned (rejected) response distributions. In this work, we extend this divergence-based perspective to general alignment settings, such as reinforcement learning with verifiable rewards (RLVR), where only environmental rewards are available. Within this unified framework, we propose -Group Relative Policy Optimization (-GRPO), a class of on-policy reinforcement learning, and -Hybrid Alignment Loss (-HAL), a hybrid on/off policy objectives, for general LLM alignment based on variational representation of -divergences. We provide theoretical guarantees that these classes of objectives improve the average reward after alignment. Empirically, we validate our framework on both RLVR (Math Reasoning) and PA tasks (Safety Alignment), demonstrating superior performance and flexibility compared to current methods.
Paper Structure (36 sections, 5 theorems, 57 equations, 2 figures, 18 tables, 1 algorithm)

This paper contains 36 sections, 5 theorems, 57 equations, 2 figures, 18 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Divergence Perspective for Preference Alignment
  3. Preliminary
  4. Notation
  5. Verifiable Reward Regime (RLVR)
  6. GRPO (On-Policy RL Aligorithm)
  7. Preference Alignment Regime
  8. Direct Aligners (off-policy) algorithms
  9. Divergence Framework
  10. Motivating the $f$-GRPO loss
  11. Divergence Estimation via Importance Sampling
  12. Estimating the Importance Sampling Term
  13. $f$-GRPO & $f$-HAL losses
  14. Main Result
  15. Canonical Link Function ($g$)
  16. ...and 21 more sections

Key Result

Theorem 4.3

Let $\theta^{(t)}_{\rm Mix}, \theta^{(t)}_{\rm RL}$ be the $t^{th}$ fixed point iterate eqn: fixed point iteration of the f-HAL & f-GRPO losses eqn: loss f-haleqn:fgrpo loss. With $G\to\infty$, the following hold almost surely: Divergence Estimation: The loss objectives satisfy: Alignment Consistency: Moreover, $\pi_{ \theta^{(t+1)}}(y|x)\propto\pi_{\rm ref}(y|x)\exp\left(\beta^{-1}h_t(x,y)\right

Figures (2)

  • Figure 1: Divergence Estimation Framework.RLVR (left): A verifiable reward signal $r(x,y)$ induces reward-aligned/unaligned distributions (above ${\cal D}^+_r$ vs. below-average reward ${\cal D}^-_r$ under the old policy), and $f$-GRPO performs on-policy alignment by estimating an $f$-divergence between these distributions. Preference alignment (right): preference data samples chosen and rejected prompt-response pairs from aligned $({\cal D}^+)$ and unaligned $({\cal D}^-)$ distributions, and direct aligners (e.g., FDO) optimize the variational $f$-divergence objective between $({\cal D}^+,{\cal D}^-)$. Hybrid (bottom):$f$-HAL combines both information sources, interpolating between off-policy preference alignment and on-policy RLVR.
  • Figure 2: Latent-space separation (Bhattacharyya distance $D_B$) between safe and harmful prompt clusters before and after alignment with $f$-HAL (Jensen--Shannon divergence) on Qwen-7B-Base. Compared to the base model, all aligned variants increase separation. The on-policy method ($f$-GRPO, $\lambda{=}0$) yields weaker separation than the hybrid ($f$-HAL, $\lambda{=}0.5$) and the off-policy supervised objective (FDO, $\lambda{=}1$).

Theorems & Definitions (13)

  • Definition 4.1: Reward-Aligned Distributions
  • Theorem 4.3
  • Theorem 4.4
  • Definition 1.1: $f$-Divergence
  • Definition 1.2: Convex Conjugate
  • Lemma 1.3: Variational Representation & Optimality
  • proof
  • Lemma 2.1: Comonotone Covariance Inequality
  • proof
  • Lemma 2.2: Monotone Reweighting Increases the Mean
  • ...and 3 more