Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving LLM General Preference Alignment via Optimistic Online Mirror Descent

Yuheng Zhang, Dian Yu, Tao Ge, Linfeng Song, Zhichen Zeng, Haitao Mi, Nan Jiang, Dong Yu

TL;DR

This work addresses the limitations of BT-based rewards in RLHF by framing LLM alignment with general preferences as a two-player zero-sum game and solving for a Nash policy. It introduces Optimistic Nash Policy Optimization (ONPO), which embeds optimistic online mirror descent into a self-play loop and uses a predictor to achieve a $DualGap(\bar{\pi})$ bound of $O(1/T)$, improving over previous $O(1/\sqrt{T})$ guarantees. The method offers a practical, data-efficient loss-based implementation that does not require estimating $\mathbb{P}(y \succ \pi_t)$, and it demonstrates strong empirical gains across AlpacaEval 2.0, Arena-Hard, MT-Bench, and academic benchmarks. The results suggest that ONPO provides robust general-preference alignment without BT-reward modeling, with potential extensions to multi-turn CMDP settings and active data collection strategies.

Abstract

Reinforcement learning from human feedback (RLHF) has demonstrated remarkable effectiveness in aligning large language models (LLMs) with human preferences. Many existing alignment approaches rely on the Bradley-Terry (BT) model assumption, which assumes the existence of a ground-truth reward for each prompt-response pair. However, this assumption can be overly restrictive when modeling complex human preferences. In this paper, we drop the BT model assumption and study LLM alignment under general preferences, formulated as a two-player game. Drawing on theoretical insights from learning in games, we integrate optimistic online mirror descent into our alignment framework to approximate the Nash policy. Theoretically, we demonstrate that our approach achieves an $O(T^{-1})$ bound on the duality gap, improving upon the previous $O(T^{-1/2})$ result. More importantly, we implement our method and show through experiments that it outperforms state-of-the-art RLHF algorithms across multiple representative benchmarks.

Improving LLM General Preference Alignment via Optimistic Online Mirror Descent

TL;DR

This work addresses the limitations of BT-based rewards in RLHF by framing LLM alignment with general preferences as a two-player zero-sum game and solving for a Nash policy. It introduces Optimistic Nash Policy Optimization (ONPO), which embeds optimistic online mirror descent into a self-play loop and uses a predictor to achieve a bound of , improving over previous guarantees. The method offers a practical, data-efficient loss-based implementation that does not require estimating , and it demonstrates strong empirical gains across AlpacaEval 2.0, Arena-Hard, MT-Bench, and academic benchmarks. The results suggest that ONPO provides robust general-preference alignment without BT-reward modeling, with potential extensions to multi-turn CMDP settings and active data collection strategies.

Abstract

Reinforcement learning from human feedback (RLHF) has demonstrated remarkable effectiveness in aligning large language models (LLMs) with human preferences. Many existing alignment approaches rely on the Bradley-Terry (BT) model assumption, which assumes the existence of a ground-truth reward for each prompt-response pair. However, this assumption can be overly restrictive when modeling complex human preferences. In this paper, we drop the BT model assumption and study LLM alignment under general preferences, formulated as a two-player game. Drawing on theoretical insights from learning in games, we integrate optimistic online mirror descent into our alignment framework to approximate the Nash policy. Theoretically, we demonstrate that our approach achieves an bound on the duality gap, improving upon the previous result. More importantly, we implement our method and show through experiments that it outperforms state-of-the-art RLHF algorithms across multiple representative benchmarks.

Paper Structure

This paper contains 35 sections, 2 theorems, 28 equations, 1 figure, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Organization.
  4. Related Work
  5. Reward-Based RLHF.
  6. RLHF with General Preferences.
  7. Learning in Games.
  8. Preliminary
  9. Problem Setup.
  10. General Preferences.
  11. Nash Policies and Duality Gap.
  12. Algorithm
  13. Self-play Algorithm with OMD Update
  14. Optimistic Nash Policy Optimization
  15. Implementation of ONPO
  16. ...and 20 more sections

Key Result

Theorem 4.1

Let $D=\max_{\pi} \mathrm{KL}(\pi \Vert \pi_1)$ and $\bar{\pi}=\frac{1}{T}\sum_{t=1}^T \pi_t$. Self-play algorithm in Eq. eq:single_update with $\eta=\sqrt{\frac{D}{T}}$ satisfies:

Figures (1)

  • Figure 1: Performance of ONPO with different values of $\eta$ on Arena-Hard and AlpacaEval 2.0. ONPO consistently outperforms the best baseline, which achieves a win rate of 25.3 on Arena-Hard and 35.3 on AlpacaEval, respectively.

Theorems & Definitions (5)

  • Definition 3.1: General Preference Oracle
  • Theorem 4.1
  • Theorem 4.2
  • proof
  • proof