Rapid Learning in Constrained Minimax Games with Negative Momentum

TL;DR

The paper addresses the slow or non-last-iterate convergence of classic minimax solvers in constrained settings by introducing negative momentum and a Restarting Aggregated Momentum framework. It extends momentum techniques to both online regret-based (RM/ RM+) and online convex optimization (FTRL/OMD) paradigms, and further adapts them to extensive-form games via CFR-style regret decomposition and dilated distance generating functions, yielding MoMWU, MoRM+, MoCFR+, and DMoGDA variants. The authors prove convergence guarantees under entropy regularization and demonstrate exponentially fast convergence to regularized equilibria, with controllable duality gaps, while empirical results on NFGs and EF games show substantial improvements in exploitability and convergence speed over SOTA baselines. The work advances constrained minimax optimization by providing a robust momentum-based toolkit that scales to large and complex game structures, with practical implications for robust learning, game AI, and equilibrium computation.

Abstract

In this paper, we delve into the utilization of the negative momentum technique in constrained minimax games. From an intuitive mechanical standpoint, we introduce a novel framework for momentum buffer updating, which extends the findings of negative momentum from the unconstrained setting to the constrained setting and provides a universal enhancement to the classic game-solver algorithms. Additionally, we provide theoretical guarantee of convergence for our momentum-augmented algorithms with entropy regularizer. We then extend these algorithms to their extensive-form counterparts. Experimental results on both Normal Form Games (NFGs) and Extensive Form Games (EFGs) demonstrate that our momentum techniques can significantly improve algorithm performance, surpassing both their original versions and the SOTA baselines by a large margin.

Figures (5)

• Figure 1: The mechanic dynamic of a particle with different force. In (a), the particle situated within a curl force diverges from the equilibrium. In (b), the augmented friction results in a reduction of the particle's velocity, thereby dampening oscillations and facilitating eventual convergence.
• Figure 2: The trajectory plots depict the MoMWU algorithm under varying negative momentum coefficients $\beta$ and intervals $k$ in Bias RPS. The initial strategy is set to $(\boldsymbol{x}, \boldsymbol{y}) = \left(\frac{1}{3}, \frac{1}{3}, \frac{1}{3}\right)$. The equilibrium strategy is denoted by a black point, whereas the blue and red points symbolize the starting and ending points of the trajectory. The learning rate $\eta$ is fixed at 2 in this context.
• Figure 3: The performance evaluation of the momentum variants and other baseline algorithms in NFGs. In all plots, the x-axis represents the number of iterations for each algorithm, while the y-axis, presented on a logarithmic scale, illustrates the exploitability.
• Figure 4: Evaluating the momentum variants and baselines in EFGs. Results are arranged by game sizes. The first three games use a logarithmic x-axis for clearer presentation.
• Figure 5: The last-iterate convergence results of DMoGDA with different parameters k, in Kuhn Poker (left) and Leduc Poker (right).

Theorems & Definitions (13)

• Proposition 1
• Theorem 2
• Theorem 3
• Theorem 4
• Lemma 5
• proof : Proof of Proposition \ref{['prop: unconstrained NM']}
• proof : Proof of Theorem \ref{['thrm: convergence of MoMWU']}
• Lemma 6
• proof
• proof : Proof of Theorem \ref{['thrm: DualityGap of MoMWU']}