Convergence of Regret Matching in Potential Games and Constrained Optimization

TL;DR

This work addresses the convergence behavior of regret matching and its variants in the nonconvex setting of potential games and constrained optimization over a product of simplices. It develops a unified analysis linking regret accumulation to first-order optimality via the KKT gap, proving that RM+ achieves an $ε$-KKT point in $O_ε(1/ε^4)$ iterations, with improvements to $O_ε(1/ε^2)$ when per-simplex regrets are bounded, and showing that discounted RM+ attains $O_1/ε^2$-type rates in potential games. A key contrast is established with RM, which can require exponential time in two-player identical-interest games to reach an approximate NE, highlighting a fundamental separation between RM and RM+. The results further demonstrate fast convergence to coarse correlated equilibria and, in potential games, to Nash equilibria, and extend to multi-simplex constrained optimization with both alternating and simultaneous updates. The findings provide theoretical justification for using RM+ as a fast, parameter-free first-order optimizer in constrained settings, while clarifying inherent limits of RM.

Abstract

Regret matching (RM) -- and its modern variants -- is a foundational online algorithm that has been at the heart of many AI breakthrough results in solving benchmark zero-sum games, such as poker. Yet, surprisingly little is known so far in theory about its convergence beyond two-player zero-sum games. For example, whether regret matching converges to Nash equilibria in potential games has been an open problem for two decades. Even beyond games, one could try to use RM variants for general constrained optimization problems. Recent empirical evidence suggests that they -- particularly regret matching$^+$ (RM$^+$) -- attain strong performance on benchmark constrained optimization problems, outperforming traditional gradient descent-type algorithms. We show that RM$^+$ converges to an $ε$-KKT point after $O_ε(1/ε^4)$ iterations, establishing for the first time that it is a sound and fast first-order optimizer. Our argument relates the KKT gap to the accumulated regret, two quantities that are entirely disparate in general but interact in an intriguing way in our setting, so much so that when regrets are bounded, our complexity bound improves all the way to $O_ε(1/ε^2)$. From a technical standpoint, while RM$^+$ does not have the usual one-step improvement property in general, we show that it does in a certain region that the algorithm will quickly reach and remain in thereafter. In sharp contrast, our second main result establishes a lower bound: RM, with or without alternation, can take an exponential number of iterations to reach a crude approximate solution even in two-player potential games. This represents the first worst-case separation between RM and RM$^+$. Our lower bound shows that convergence to coarse correlated equilibria in potential games is exponentially faster than convergence to Nash equilibria.

Figures (2)

• Figure 1: Illustration of our main results: $\texttt{RM}^+$ always converges fast to a KKT point while $\texttt{RM}$ can take exponential time even in two-player identical-interest games, constructed in \ref{['sec:RM']}.
• Figure 2: The example corresponding to \ref{['prop:4cycle']}, demonstrating that having zero regret, let alone sublinear, has no implications concerning convergence in terms of KKT gap.

Theorems & Definitions (72)

• Theorem 1.1
• Theorem 1.2
• Theorem 1.3
• Corollary 1.4
• Definition 2.1
• Definition 2.2: Potential game
• Proposition 2.3: Zinkevich07:RegretFarina21:Faster
• Proposition 3.1: Under concavity, no-regret implies convergence
• Proposition 3.2
• Lemma 3.2: One-step improvement for $\RMplus$