Nash Policy Gradient: A Policy Gradient Method with Iteratively Refined Regularization for Finding Nash Equilibria

TL;DR

This work tackles finding Nash equilibria in imperfect-information, two-player zero-sum games by replacing decaying regularization with an iteratively refined, fixed-strong regularizer. The authors introduce a regularized VI operator $M$ that maps a reference policy to the solution of a regularized VI, and prove that iteratively applying $M$ yields strictly decreasing Bregman divergence to any Nash equilibrium and convergence to a Nash equilibrium without assuming uniqueness. Building on this theory, they derive Nash Policy Gradient (NashPG), a practical, RL-friendly algorithm that operates with behavioral strategies and KL regularization, updating only current and reference policies in outer iterations. Empirically, NashPG matches or outperforms baselines on standard benchmarks and scales to large domains like Battleship and No-Limit Hold’em, achieving higher Elo ratings and robust convergence. The method offers a simple, modular addition to policy-gradient pipelines with strong theoretical and empirical performance, suggesting substantial practical impact for scalable MARL in imperfect-information settings.

Abstract

Finding Nash equilibria in imperfect-information games remains a central challenge in multi-agent reinforcement learning. While regularization-based methods have recently achieved last-iteration convergence to a regularized equilibrium, they require the regularization strength to shrink toward zero to approximate a Nash equilibrium, often leading to unstable learning in practice. Instead, we fix the regularization strength at a large value for robustness and achieve convergence by iteratively refining the reference policy. Our main theoretical result shows that this procedure guarantees strictly monotonic improvement and convergence to an exact Nash equilibrium in two-player zero-sum games, without requiring a uniqueness assumption. Building on this framework, we develop a practical algorithm, Nash Policy Gradient (NashPG), which preserves the generalizability of policy gradient methods while relying solely on the current and reference policies. Empirically, NashPG achieves comparable or lower exploitability than prior model-free methods on classic benchmark games and scales to large domains such as Battleship and No-Limit Texas Hold'em, where NashPG consistently attains higher Elo ratings.

Figures (6)

• Figure 1: Exploitability curves for our benchmarks. NashPG consistently achieves lower final exploitability across all tested environments except Kuhn Poker. Error bars show standard deviation over 4 runs.
• Figure 2: Exploitability in real-world games. Battleship and Texas Hold'em show negative values (theoretically impossible), highlighting the limitations of exploitability-based evaluation under limited computational budgets in complex domains.
• Figure 3: Elo ratings in large real-world games. NashPG achieves consistently higher final Elo ratings across complex domains where exploitability is unreliable. Error bars denote standard deviation over 4 runs.
• Figure 4: Exploitability under annealing different initial values for $\alpha$ (optionally annealing $\eta$). In all cases, exploitability diverges as $\alpha$ decreases.
• Figure 5: Maximum GPU memory usage during training across environments and algorithms.

Theorems & Definitions (26)

• theorem 1: Informal
• definition 1: Variational Inequality Problem
• theorem 2: Nash-VI Equivalence
• definition 2: Regularized VI Operator
• lemma 1: Distance Non-increase Property
• lemma 2: Fixed Point Characterization
• lemma 3: Continuity of $\gM$
• theorem 3: Convergence of Iterative $\gM$
• definition 3: $L$-smooth function
• definition 4: $\mu$-strong convex function