Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuning

Yu Luo, Shuo Han, Yihan Hu, Dong Li, Jianye Hao

TL;DR

R^2VPO introduces a ratio-variance regularization to replace hard clipping in TRPO-style policy optimization for LLM fine-tuning. It derives a principled trust-region surrogate by linking the divergence between current and off-policy policies to the variance of the policy ratio $\rho_t(\theta)$ via a Jensen–Shannon divergence approximation, and implements a primal–dual objective with a quadratic penalty on $(\rho_t(\theta)-1)^2$. Empirically, R^2VPO delivers up to $17\%$ average gains and requires about $50\%$ fewer rollouts across multiple LLMs and math-reasoning benchmarks, with strong performance in both on-policy stability and off-policy data reuse. The approach provides a robust, critic-free framework for stable, data-efficient RL-based LLM alignment and shows promise for broader decision-making tasks with large action spaces.

Abstract

On-policy reinforcement learning (RL), particularly Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), has become the dominant paradigm for fine-tuning large language models (LLMs). While policy ratio clipping stabilizes training, this heuristic hard constraint incurs a fundamental cost: it indiscriminately truncates gradients from high-return yet high-divergence actions, suppressing rare but highly informative "eureka moments" in complex reasoning. Moreover, once data becomes slightly stale, hard clipping renders it unusable, leading to severe sample inefficiency. In this work, we revisit the trust-region objective in policy optimization and show that explicitly constraining the \emph{variance (second central moment) of the policy ratio} provides a principled and smooth relaxation of hard clipping. This distributional constraint stabilizes policy updates while preserving gradient signals from valuable trajectories. Building on this insight, we propose $R^2VPO$ (Ratio-Variance Regularized Policy Optimization), a novel primal-dual framework that supports stable on-policy learning and enables principled off-policy data reuse by dynamically reweighting stale samples rather than discarding them. We extensively evaluate $R^2VPO$ on fine-tuning state-of-the-art LLMs, including DeepSeek-Distill-Qwen-1.5B and the openPangu-Embedded series (1B and 7B), across challenging mathematical reasoning benchmarks. Experimental results show that $R^2VPO$ consistently achieves superior asymptotic performance, with average relative gains of up to 17% over strong clipping-based baselines, while requiring approximately 50% fewer rollouts to reach convergence. These findings establish ratio-variance control as a promising direction for improving both stability and data efficiency in RL-based LLM alignment.

Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuning

TL;DR

R^2VPO introduces a ratio-variance regularization to replace hard clipping in TRPO-style policy optimization for LLM fine-tuning. It derives a principled trust-region surrogate by linking the divergence between current and off-policy policies to the variance of the policy ratio via a Jensen–Shannon divergence approximation, and implements a primal–dual objective with a quadratic penalty on . Empirically, R^2VPO delivers up to average gains and requires about fewer rollouts across multiple LLMs and math-reasoning benchmarks, with strong performance in both on-policy stability and off-policy data reuse. The approach provides a robust, critic-free framework for stable, data-efficient RL-based LLM alignment and shows promise for broader decision-making tasks with large action spaces.

Abstract

On-policy reinforcement learning (RL), particularly Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), has become the dominant paradigm for fine-tuning large language models (LLMs). While policy ratio clipping stabilizes training, this heuristic hard constraint incurs a fundamental cost: it indiscriminately truncates gradients from high-return yet high-divergence actions, suppressing rare but highly informative "eureka moments" in complex reasoning. Moreover, once data becomes slightly stale, hard clipping renders it unusable, leading to severe sample inefficiency. In this work, we revisit the trust-region objective in policy optimization and show that explicitly constraining the \emph{variance (second central moment) of the policy ratio} provides a principled and smooth relaxation of hard clipping. This distributional constraint stabilizes policy updates while preserving gradient signals from valuable trajectories. Building on this insight, we propose (Ratio-Variance Regularized Policy Optimization), a novel primal-dual framework that supports stable on-policy learning and enables principled off-policy data reuse by dynamically reweighting stale samples rather than discarding them. We extensively evaluate on fine-tuning state-of-the-art LLMs, including DeepSeek-Distill-Qwen-1.5B and the openPangu-Embedded series (1B and 7B), across challenging mathematical reasoning benchmarks. Experimental results show that consistently achieves superior asymptotic performance, with average relative gains of up to 17% over strong clipping-based baselines, while requiring approximately 50% fewer rollouts to reach convergence. These findings establish ratio-variance control as a promising direction for improving both stability and data efficiency in RL-based LLM alignment.
Paper Structure (21 sections, 2 theorems, 12 equations, 3 figures, 1 table, 1 algorithm)

This paper contains 21 sections, 2 theorems, 12 equations, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary
  3. RL Formulation for LLMs.
  4. Group Relative Policy Optimization.
  5. Ratio-Variance Regularized Policy Optimization
  6. From JS Divergence to Policy Ratio Variance
  7. Primal-Dual Formulation
  8. Practical Implementation
  9. On-Policy Training.
  10. Off-Policy Training.
  11. Experiment
  12. Experimental Setup
  13. Training Datasets.
  14. Base Models & Baselines.
  15. Benchmarks.
  16. ...and 6 more sections

Key Result

Lemma 3.1

Let $\rho_t(\theta) = \pi_\theta(a_t|s_t) / \pi_{\text{off}}(a_t|s_t)$ be the policy ratio. For a policy $\pi_\theta$ sufficiently close to $\pi_{\text{off}}$, the expected Jensen-Shannon divergence can be approximated by the second-order moment (variance) of the policy ratio:

Figures (3)

  • Figure 1: R$^2$VPO achieves a 17% average performance gain while requiring 50% fewer data rollouts compared to strong baselines on DeepSeek-Distill-Qwen-1.5B. The main bar chart details the Pass@1 accuracy across five math benchmarks, where the off-policy variant consistently demonstrates superior asymptotic performance. The inset visualizes the training dynamics, highlighting a total improvement of 119% over the base model and rapid convergence that matches the baselines' peak performance in significantly fewer steps.
  • Figure 2: Visualization of Policy Ratio Dynamics. Hard clipping indiscriminately truncates high-value exploration, while variance regularization ensures robust stability. (a) The Cost of Clipping: As shown by the colored points ($|\rho_t - 1| > \epsilon$), standard clipping disproportionately affects low-probability tokens even with positive advantage. This effectively suppresses high-uncertainty yet critical "eureka moments" in reasoning. (b) & (c) The Variance Solution: The proposed variance proxy (orange) demonstrates tight alignment with the true JS divergence (blue). Unlike KL-divergence, it maintains numerical stability even when $\pi_{\text{off}} \to 0$, preventing optimization collapse.
  • Figure 3: Training Dynamics and Evaluation Results. The learning curves illustrate the evolution of test accuracy and reward throughout the training process. R$^2$VPO (orange lines) consistently achieves faster convergence and higher asymptotic performance compared to baseline methods, particularly in the off-policy setting where data efficiency is maximized.

Theorems & Definitions (2)

  • Lemma 3.1: Approximation of JS Divergence
  • Theorem 3.1: R$^2$VPO Primal-Dual Objective