Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees

Yongtao Wu, Luca Viano, Yihang Chen, Zhenyu Zhu, Kimon Antonakopoulos, Quanquan Gu, Volkan Cevher

TL;DR

This work reframes multi-step RLHF as a two-player constant-sum Markov game to capture intermediate and non-transitive human preferences, moving beyond Bradley-Terry assumptions and terminal-only signals. It introduces Optimistic Multi-step Preference Optimization (OMPO), which applies optimistic online mirror descent to the occupancy-measure formulation, yielding an $O(ε^{-1})$ policy-update convergence to an $ε$-approximate Nash equilibrium and a projection-free, practical implementation. Theoretical results are complemented by empirical validation on multi-turn conversation benchmarks (MT-bench-101) and math reasoning datasets, where OMPO and its MPO variant outperform several baselines and adapt to intermediate versus terminal rewards. The approach enables per-turn preference learning with general, non-transitive signals while providing convergence guarantees and scalable implementation, potentially impacting real-world alignment in complex dialogue and reasoning tasks.

Abstract

Reinforcement Learning from Human Feedback (RLHF) has been highly successful in aligning large language models with human preferences. While prevalent methods like DPO have demonstrated strong performance, they frame interactions with the language model as a bandit problem, which limits their applicability in real-world scenarios where multi-turn conversations are common. Additionally, DPO relies on the Bradley-Terry model assumption, which does not adequately capture the non-transitive nature of human preferences. In this paper, we address these challenges by modeling the alignment problem as a two-player constant-sum Markov game, where each player seeks to maximize their winning rate against the other across all steps of the conversation. Our approach Optimistic Multi-step Preference Optimization (OMPO) is built upon the optimistic online mirror descent algorithm~\citep{rakhlin2013online,joulani17a}. Theoretically, we provide a rigorous analysis for the convergence of OMPO and show that OMPO requires $\mathcal{O}(ε^{-1})$ policy updates to converge to an $ε$-approximate Nash equilibrium. We also validate the effectiveness of our method on multi-turn conversations dataset and math reasoning dataset.

Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees

TL;DR

This work reframes multi-step RLHF as a two-player constant-sum Markov game to capture intermediate and non-transitive human preferences, moving beyond Bradley-Terry assumptions and terminal-only signals. It introduces Optimistic Multi-step Preference Optimization (OMPO), which applies optimistic online mirror descent to the occupancy-measure formulation, yielding an policy-update convergence to an -approximate Nash equilibrium and a projection-free, practical implementation. Theoretical results are complemented by empirical validation on multi-turn conversation benchmarks (MT-bench-101) and math reasoning datasets, where OMPO and its MPO variant outperform several baselines and adapt to intermediate versus terminal rewards. The approach enables per-turn preference learning with general, non-transitive signals while providing convergence guarantees and scalable implementation, potentially impacting real-world alignment in complex dialogue and reasoning tasks.

Abstract

Reinforcement Learning from Human Feedback (RLHF) has been highly successful in aligning large language models with human preferences. While prevalent methods like DPO have demonstrated strong performance, they frame interactions with the language model as a bandit problem, which limits their applicability in real-world scenarios where multi-turn conversations are common. Additionally, DPO relies on the Bradley-Terry model assumption, which does not adequately capture the non-transitive nature of human preferences. In this paper, we address these challenges by modeling the alignment problem as a two-player constant-sum Markov game, where each player seeks to maximize their winning rate against the other across all steps of the conversation. Our approach Optimistic Multi-step Preference Optimization (OMPO) is built upon the optimistic online mirror descent algorithm~\citep{rakhlin2013online,joulani17a}. Theoretically, we provide a rigorous analysis for the convergence of OMPO and show that OMPO requires policy updates to converge to an -approximate Nash equilibrium. We also validate the effectiveness of our method on multi-turn conversations dataset and math reasoning dataset.

Paper Structure

This paper contains 40 sections, 9 theorems, 108 equations, 1 figure, 8 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Problem setting: Multi-step RLHF as two-player Markov games
  3. Notation
  4. Problem formulation of multi-step RLHF
  5. Useful facts in Markov games
  6. The occupancy measure view
  7. Algorithm and convergence guarantees
  8. Convergence guarantees of optimistic multi-step preference optimization (OMPO)
  9. Efficient implementation
  10. Approximating the value function updates
  11. Approximating the policy update
  12. Experiments
  13. Tabular experiment
  14. Experiment on multi-turn conversation dataset
  15. Experiment on math reasoning dataset
  16. ...and 25 more sections

Key Result

Theorem 4

Consider alg:forb and let us assume that the occupancy measure of the reference policy $d^1$ is uniformly lower bounded by $\underline{d}$. Moreover, let $\mathbb{D}$ be $1/\lambda$ strongly convex, i.e. $\mathbb{D}(p||q) \geq \frac{\left\lVert p-q\right\rVert^2_1}{2\lambda}$. Then, by setting $T =

Figures (1)

  • Figure 1: (a): Results in the tabular experiments. Curves are averages across $10$ different randomly generated environments. The error bars report one standard deviation. (b): Result of OMPO on the MT-bench-101 dataset; (c) Winning rate against the base model with different approximations for the $Q$ functions. When optimizing $a_h$ at the $h$ step, only considering the preference of $s_{h}$ is sufficient compared to using $s_{h},\dots,s_{H+1}$.

Theorems & Definitions (27)

  • Example 1: Single-step alignment
  • Example 2: Chain-of-thought reasoning alignment
  • Example 3: Multi-turn conversation alignment
  • Remark 1
  • Definition 1: $\epsilon$-approximate Nash equilibrium
  • Theorem 4: Convergence of OMPO
  • Theorem 5
  • Theorem 6
  • Theorem 7
  • Lemma 1: Value difference lemma (Adapted from kakade2002approximately)
  • ...and 17 more