Multiplayer Nash Preference Optimization

TL;DR

MNPO generalizes NLHF to an $n$-player framework where each policy competes against a population of opponents and stays anchored to a reference model. It combines Plackett-Luce reward learning with time-dependent opponent mixtures to stabilize training and achieve an $\epsilon$-approximate Nash equilibrium in homogeneous settings, while HT-MNPO addresses heterogeneous evaluators. The approach unifies existing RLHF methods as special cases and demonstrates superior performance across instruction-following, knowledge, and math/coding benchmarks, even under diverse preferences. The work provides a principled, scalable foundation for aligning LLMs to complex, non-transitive human preferences and broadens the practical applicability of equilibrium-based RLHF.

Abstract

Reinforcement learning from human feedback (RLHF) has emerged as the standard paradigm for aligning large language models with human preferences. However, reward-based methods built on the Bradley-Terry assumption struggle to capture the non-transitive and heterogeneous nature of real-world preferences. To address this, recent studies have reframed alignment as a two-player Nash game, giving rise to Nash learning from human feedback (NLHF). While this perspective has inspired algorithms such as INPO, ONPO, and EGPO with strong theoretical and empirical guarantees, they remain fundamentally restricted to two-player interactions, creating a single-opponent bias that fails to capture the full complexity of realistic preference structures. This work introduces Multiplayer Nash Preference Optimization (MNPO), a novel framework that generalizes NLHF to the multiplayer regime. It formulates alignment as an n-player game, where each policy competes against a population of opponents while being regularized toward a reference model. We demonstrate that MNPO inherits the equilibrium guarantees of two-player methods while enabling richer competitive dynamics and improved coverage of diverse preference structures. Comprehensive empirical evaluation shows that MNPO consistently outperforms existing NLHF baselines on instruction-following benchmarks, achieving superior alignment quality under heterogeneous annotator conditions and mixed-policy evaluation scenarios. Together, these results establish MNPO as a principled and scalable framework for aligning LLMs with complex, non-transitive human preferences. Code is available at https://github.com/smiles724/MNPO.

Theorems & Definitions (5)

• Lemma 1
• Proposition 1
• proof
• proof
• Proposition 2