Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerating Nash Learning from Human Feedback via Mirror Prox

Daniil Tiapkin, Daniele Calandriello, Denis Belomestny, Eric Moulines, Alexey Naumov, Kashif Rasul, Michal Valko, Pierre Menard

TL;DR

This work reframes human feedback learning as a symmetric two-player preference game and introduces Nash Mirror Prox (Nash-MP) to compute the $β$-regularized Nash equilibrium with strong convergence guarantees. Nash-MP achieves last-iterate linear convergence in KL-divergence and suboptimality, with rates that scale as $(1+2β)^{-N/2}/β$ and are independent of action-space size, outperforming prior NLHF methods. An approximate, policy-gradient-based variant enables practical deep-learning deployment and practical LLM fine-tuning, with theoretical guarantees linking approximation error to remaining suboptimality. Empirically, Nash-MP demonstrates competitive performance in matrix games and superior results in LLM alignment tasks compared to several baselines, supporting its potential to improve human-preference alignment while maintaining proximity to reference policies.

Abstract

Traditional Reinforcement Learning from Human Feedback (RLHF) often relies on reward models, frequently assuming preference structures like the Bradley-Terry model, which may not accurately capture the complexities of real human preferences (e.g., intransitivity). Nash Learning from Human Feedback (NLHF) offers a more direct alternative by framing the problem as finding a Nash equilibrium of a game defined by these preferences. In this work, we introduce Nash Mirror Prox ($\mathtt{Nash-MP}$), an online NLHF algorithm that leverages the Mirror Prox optimization scheme to achieve fast and stable convergence to the Nash equilibrium. Our theoretical analysis establishes that Nash-MP exhibits last-iterate linear convergence towards the $β$-regularized Nash equilibrium. Specifically, we prove that the KL-divergence to the optimal policy decreases at a rate of order $(1+2β)^{-N/2}$, where $N$ is a number of preference queries. We further demonstrate last-iterate linear convergence for the exploitability gap and uniformly for the span semi-norm of log-probabilities, with all these rates being independent of the size of the action space. Furthermore, we propose and analyze an approximate version of Nash-MP where proximal steps are estimated using stochastic policy gradients, making the algorithm closer to applications. Finally, we detail a practical implementation strategy for fine-tuning large language models and present experiments that demonstrate its competitive performance and compatibility with existing methods.

Accelerating Nash Learning from Human Feedback via Mirror Prox

TL;DR

This work reframes human feedback learning as a symmetric two-player preference game and introduces Nash Mirror Prox (Nash-MP) to compute the -regularized Nash equilibrium with strong convergence guarantees. Nash-MP achieves last-iterate linear convergence in KL-divergence and suboptimality, with rates that scale as and are independent of action-space size, outperforming prior NLHF methods. An approximate, policy-gradient-based variant enables practical deep-learning deployment and practical LLM fine-tuning, with theoretical guarantees linking approximation error to remaining suboptimality. Empirically, Nash-MP demonstrates competitive performance in matrix games and superior results in LLM alignment tasks compared to several baselines, supporting its potential to improve human-preference alignment while maintaining proximity to reference policies.

Abstract

Traditional Reinforcement Learning from Human Feedback (RLHF) often relies on reward models, frequently assuming preference structures like the Bradley-Terry model, which may not accurately capture the complexities of real human preferences (e.g., intransitivity). Nash Learning from Human Feedback (NLHF) offers a more direct alternative by framing the problem as finding a Nash equilibrium of a game defined by these preferences. In this work, we introduce Nash Mirror Prox (), an online NLHF algorithm that leverages the Mirror Prox optimization scheme to achieve fast and stable convergence to the Nash equilibrium. Our theoretical analysis establishes that Nash-MP exhibits last-iterate linear convergence towards the -regularized Nash equilibrium. Specifically, we prove that the KL-divergence to the optimal policy decreases at a rate of order , where is a number of preference queries. We further demonstrate last-iterate linear convergence for the exploitability gap and uniformly for the span semi-norm of log-probabilities, with all these rates being independent of the size of the action space. Furthermore, we propose and analyze an approximate version of Nash-MP where proximal steps are estimated using stochastic policy gradients, making the algorithm closer to applications. Finally, we detail a practical implementation strategy for fine-tuning large language models and present experiments that demonstrate its competitive performance and compatibility with existing methods.

Paper Structure

This paper contains 70 sections, 23 theorems, 267 equations, 2 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Setting
  4. Preference game.
  5. Von Neumann winner.
  6. Regularized preference game.
  7. Additional notations.
  8. Nash Mirror Prox
  9. Algorithm Description
  10. Connection with Proximal Point method.
  11. Theoretical Guarantees
  12. Comparison to existing results.
  13. Approximate Nash Mirror Prox
  14. Algorithm
  15. Theoretical Guarantees
  16. ...and 55 more sections

Key Result

Theorem 1

Assume $\beta \leq 1/2$. After $K$ iterations of $\mathtt{NashMP}$ with a learning rate $\eta \leq 2\beta$ an initial policy $\pi_0 = \pi^{\mathrm{ref}}$, the suboptimality gap and KL-divergence to the optimal solution satisfy At the same time, the algorithm enjoys linear rates to the optimal solution in the span semi-norm

Figures (2)

  • Figure 1: Comparison of $\mathtt{NashMP}$ (in red) with baseline methods across different optimization horizons $K \in \{10^3, 10^4, 5 \times 10^4\}$. Our method consistently achieves lower suboptimality as the optimization horizon increases. Suboptimality is averaged over 10 random seeds; shaded regions indicate one standard deviation.
  • Figure 2: Effect of the soft update parameter $\kappa$ on the performance of $\mathtt{NashMP}$ across optimization horizons $K \in \{10^3, 10^4, 5 \times 10^4\}$. Smaller $\kappa$ values are needed for effective optimization as $K$ increases. Suboptimality is averaged over 10 random seeds; shaded regions indicate one standard deviation.

Theorems & Definitions (42)

  • Theorem 1
  • Corollary 1
  • Theorem 2
  • Lemma 1
  • Theorem : Restatement of Theorem \ref{['th:nash_mp_convergence']}
  • proof
  • Corollary : Restatement of Corollary \ref{['cor:nash_mp_complexity']}
  • proof
  • Lemma 2
  • proof
  • ...and 32 more