Latent Adversarial Regularization for Offline Preference Optimization

TL;DR

GANPO introduces latent-space adversarial regularization to offline preference optimization, addressing semantic gaps left by token-space constraints. By anchoring latent representations to a reference model and employing a quad-gan discriminator setup, GANPO provides dense structural feedback that preserves latent geometry during optimization. Across diverse model architectures and tasks, GANPO yields consistent improvements over DPO and SimPO with modest computational overhead and enhanced robustness to noise and distributional shift, while maintaining downstream performance. The approach offers a stable, reference-anchored method for more robust alignment of LLMs with human preferences, with potential extensions to online and cross-modal settings.

Abstract

Learning from human feedback typically relies on preference optimization that constrains policy updates through token-level regularization. However, preference optimization for language models is particularly challenging because token-space similarity does not imply semantic or behavioral similarity. To address this challenge, we leverage latent-space regularization for language model preference optimization. We introduce GANPO, which achieves latent-space regularization by penalizing divergence between the internal representations of a policy model and a reference model. Given that latent representations are not associated with explicit probability densities, we adopt an adversarial approach inspired by GANs to minimize latent-space divergence. We integrate GANPO as a regularizer into existing offline preference optimization objectives. Experiments across multiple model architectures and tasks show consistent improvements from latent-space regularization. Further, by comparing GANPO-induced inferential biases with those from token-level regularization, we find that GANPO provides more robust structural feedback under distributional shift and noise while maintaining comparable downstream performance with minor computational overhead.

Figures (10)

• Figure 1: Comparison between DPO and GANPO. Offline preference optimization methods (e.g., DPO) optimize an implicit reward defined by preference data. GANPO augments this objective with a latent-space discriminator, whose adversarial interaction induces a regularization between the latent representation distributions of the policy model and the reference model.
• Figure 2: Latent space vs token space. Anchor ("Hi there.") is the reference point for distance measurements. Semantically similar paraphrases exhibit large token-level variation yet remain close in latent space, while semantically different phrases show smaller token changes but larger latent space differences.
• Figure 3: Robustness against entropy. Comparison of model performance across varying sampling temperatures ($T \in [0.0, 1.5]$). Unlike DPO, which relies heavily on greedy decoding for peak performance and collapses under noise, GANPO acts as a structural regularizer, effectively preserving both preference alignment and constraint satisfaction during high-entropy generation.
• Figure 4: Comparison of discriminator-based scoring and learned reward models under high-entropy generation. At elevated sampling temperatures ($T=1.5$ and $T=2.0$), the learned reward model exhibits severe reward hacking, including correlation collapse and inversion with respect to the oracle. In contrast, the discriminator maintains a strong positive correlation, demonstrating robustness to out-of-distribution generations and providing stable structural supervision in latent space.
• Figure 5: Win rate as a function of response length for DPO and GANPO.

Theorems & Definitions (4)

• Definition 1.1: Statistical Divergence
• Proposition 1.2: Relativistic Average Divergence jolicoeur2020relativistic
• Proposition 1.3
• proof