Unregularized Linear Convergence in Zero-Sum Game from Preference Feedback

TL;DR

This work addresses aligning large language models to human preferences by reframing the problem as Nash Learning from Human Feedback (NLHF), a two-player zero-sum game capturing non-transitive population preferences. It analyzes Optimistic Multiplicative Weights Update (OMWU) in NLHF without assuming NE uniqueness, proving last-iterate linear convergence after a burn-in under a full-support NE, with an instance-dependent rate governed by constants such as $oldsymbol{igvarepsilon}$, $L$, and $C_{m{P}}$. A novel marginal and subgame escape analysis yields polynomial dependence on instance constants rather than exponential, improving practicality for NLHF tasks. Theoretical results are complemented by experiments on tabular and neural policies, where OMWU demonstrates strong convergence and outperforms regularized baselines that suffer from bias or nested-optimization challenges. Overall, the paper establishes a robust, unregularized approach to NLHF with provable convergence guarantees, highlighting its potential for scalable LLM alignment.

Abstract

Aligning large language models (LLMs) with human preferences has proven effective for enhancing model capabilities, yet standard preference modeling using the Bradley-Terry model assumes transitivity, overlooking the inherent complexity of human population preferences. Nash learning from human feedback (NLHF) addresses this by framing non-transitive preferences as a two-player zero-sum game, where alignment reduces to finding the Nash equilibrium (NE). However, existing algorithms typically rely on regularization, incurring unavoidable bias when computing the duality gap in the original game. In this work, we provide the first convergence guarantee for Optimistic Multiplicative Weights Update ($\mathtt{OMWU}$) in NLHF, showing that it achieves last-iterate linear convergence after a burn-in phase whenever an NE with full support exists, with an instance-dependent linear convergence rate to the original NE, measured by duality gaps. Compared to prior results in Wei et al. (2020), we do not require the assumption of NE uniqueness. Our analysis identifies a novel marginal convergence behavior, where the probability of rarely played actions grows exponentially from exponentially small values, enabling exponentially better dependence on instance-dependent constants than prior results. Experiments corroborate the theoretical strengths of $\mathtt{OMWU}$ in both tabular and neural policy classes, demonstrating its potential for LLM applications.

Figures (5)

• Figure 1: Evolution of $\Theta_t-\Theta_{t+1}$ for ${\bm{P}}=01/2-1/2-1/201/21/2-1/20$ with initialization near $(1,0,0)$.
• Figure 2: Selected results under the tabular (left) and neural (right) policy setting.
• Figure 3: Duality gaps of OMD (regularized) applied to tabular policies, when evaluating the last-iterate policy ${\bm{\pi}}^{(t)}$ and the average-iterate policy $\frac{1}{t} \sum_{i=1}^t {\bm{\pi}}^{(i)}$.
• Figure 4: Duality gaps of different algorithms applied to tabular policies.
• Figure 5: Duality gaps of different algorithms applied to neural policies.

Theorems & Definitions (49)

• Remark 1
• Lemma 1
• Lemma 2
• Remark 2
• Definition 1: Instance-dependent constants
• Theorem 2
• Remark 3
• Remark 4
• Remark 5
• Lemma 3