Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Orthogonalized Policy Optimization:Decoupling Sampling Geometry from Optimization Geometry in RLHF

Wang Zixian

TL;DR

This work proposes a simple but structural remedy by formulating alignment as an orthogonal mirror descent problem, in which sampling geometry enters only as a linear driving force, while optimization geometry is determined independently by a mirror map.

Abstract

Recent alignment methods for large language models, including PPO, DPO, and IPO, are often presented as distinct algorithms. In this work, we show that many of these approaches implicitly conflate two fundamental and independent design choices: (i) the sampling geometry, which determines which samples dominate the gradient signal, and (ii) the optimization geometry, which determines how deviations in value are penalized. We formalize this observation by expressing alignment as the minimization of a generalized distance between policy energy and target energy, parameterized by an alpha-divergence-based sampling weight and a Bregman-divergence-based value metric. We demonstrate that the commonly used KL divergence induces an exponential penalty on unbounded value signals, leading to numerical instability and vanishing gradients in high-confidence regimes. To address this issue, we propose Orthogonalized Policy Optimization (OPO), a framework that explicitly decouples sampling geometry from optimization geometry. By combining alpha-weighted importance sampling with a chi-square-induced quadratic regularization in ratio coordinates, OPO yields a simple and well-conditioned objective with linear gradient dynamics. This formulation maintains stable optimization while preserving peak-seeking behavior and avoids gradient saturation even when model confidence is high. Our analysis positions OPO as a unifying perspective on existing alignment methods and provides a principled foundation for robust reasoning-oriented training.

Orthogonalized Policy Optimization:Decoupling Sampling Geometry from Optimization Geometry in RLHF

TL;DR

This work proposes a simple but structural remedy by formulating alignment as an orthogonal mirror descent problem, in which sampling geometry enters only as a linear driving force, while optimization geometry is determined independently by a mirror map.

Abstract

Recent alignment methods for large language models, including PPO, DPO, and IPO, are often presented as distinct algorithms. In this work, we show that many of these approaches implicitly conflate two fundamental and independent design choices: (i) the sampling geometry, which determines which samples dominate the gradient signal, and (ii) the optimization geometry, which determines how deviations in value are penalized. We formalize this observation by expressing alignment as the minimization of a generalized distance between policy energy and target energy, parameterized by an alpha-divergence-based sampling weight and a Bregman-divergence-based value metric. We demonstrate that the commonly used KL divergence induces an exponential penalty on unbounded value signals, leading to numerical instability and vanishing gradients in high-confidence regimes. To address this issue, we propose Orthogonalized Policy Optimization (OPO), a framework that explicitly decouples sampling geometry from optimization geometry. By combining alpha-weighted importance sampling with a chi-square-induced quadratic regularization in ratio coordinates, OPO yields a simple and well-conditioned objective with linear gradient dynamics. This formulation maintains stable optimization while preserving peak-seeking behavior and avoids gradient saturation even when model confidence is high. Our analysis positions OPO as a unifying perspective on existing alignment methods and provides a principled foundation for robust reasoning-oriented training.
Paper Structure (41 sections, 2 theorems, 12 equations, 2 figures)

This paper contains 41 sections, 2 theorems, 12 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Observation: Implicit Conflation of Two Design Choices.
  3. Diagnosis: The KL-Induced Exponential Penalty.
  4. Contribution: Orthogonalized Policy Optimization (OPO).
  5. Related Work
  6. Preference Optimization and RLHF.
  7. $f$-Divergences in Machine Learning.
  8. Trust-Region Methods.
  9. Theoretical Framework: Two Independent Design Axes
  10. Coordinates: Ratio Space vs. Log-Ratio Space
  11. The Generalized Alignment Objective
  12. Axis 1: Sampling Geometry via $\alpha$-Divergence
  13. Axis 2: Optimization Geometry via Divergence Choice
  14. KL-Regularized Objectives (Exponential-Type Penalties).
  15. $\chi^2$ Geometry (Quadratic Penalty in Ratio Coordinates).
  16. ...and 26 more sections

Key Result

Proposition 4.1

Let $t(y) = \pi(y)/\pi_{\mathrm{ref}}(y)$ and $v(y) = t(y) - 1$. The Pearson $\chi^2$ divergence satisfies: Therefore, a $\chi^2$ trust-region regularization yields a quadratic regularizer in the ratio coordinate $v$.

Figures (2)

  • Figure 1: Training dynamics comparing OPO ($\alpha=0.6$, $\mu=1.0$) and GSPO on Qwen3-1.7B Math RL. Solid lines show 7-step moving averages; faded lines show raw values. Both methods achieve comparable final performance, with OPO showing a slight edge.
  • Figure 2: Gradient norm comparison. OPO maintains higher gradient norms throughout training, consistent with the theoretical prediction of non-saturating gradient dynamics.

Theorems & Definitions (6)

  • Definition 3.1: Ratio Coordinates
  • Definition 3.2: Log-Ratio Coordinates
  • Proposition 4.1: Pearson $\chi^2$ induces a quadratic penalty in ratio coordinates
  • Corollary 4.2: Local quadratic form in log-ratio space
  • Definition 5.1: Orthogonality of Design Axes
  • Remark 5.2: Intuition