(Doubly) Exponential Lower Bounds for Follow the Regularized Leader in Potential Games

TL;DR

This paper studies convergence of Follow the Regularized Leader ($\texttt{FTRL}$) in potential games and shows that, in two-player settings, $\texttt{FTRL}$ can require $2^{\Omega(m \log m)}$ iterations to reach an $\varepsilon$-Nash for any permutation-invariant regularizer and learning rate $\eta^{(t)}=t^{-\\alpha}$, $\alpha\in[0,1)$. By equivalences with online mirror descent, this also implies exponential lower bounds for multiplicative weights updates, while a positive result establishes an exponential upper bound $\exp(O_{\varepsilon}(1/\varepsilon^2))$ for alternating, lazy no-regret dynamics and proves the potential property of $\texttt{FTRL}$ (nondecreasing potential). In multi-player potential games, fictitious play ($\eta\to\infty$) attains a doubly exponential lower bound, requiring $2^{\Omega(2^n)}$ iterations to converge to a Nash equilibrium. The results delineate a sharp separation between last-iterate convergence and no-regret guarantees, highlighting that $\texttt{FTRL}$ can be a poor first-order optimizer even in inverse-polynomial accuracy regimes. Overall, the paper provides tight upper and lower bounds that clarify the computational landscape of key learning dynamics in potential games and points to further questions about the regimes under which doubly-exponential behavior persists.

Abstract

Follow the regularized leader FTRL is the premier algorithm for online optimization. However, despite decades of research on its convergence in constrained optimization -- and potential games in particular -- its behavior remained hitherto poorly understood. In this paper, we establish that FTRL can take exponential time to converge to a Nash equilibrium in two-player potential games for any (permutation-invariant) regularizer and potentially vanishing learning rate. By known equivalences, this translates to an exponential lower bound for certain mirror descent counterparts, most notably multiplicative weights update. On the positive side, we establish the potential property for FTRL and obtain an exponential upper bound $\exp(O_ε(1/ε^2))$ for any no-regret dynamics executed in a lazy, alternating fashion, matching our lower bound up to factors in the exponent. Finally, in multi-player potential games, we show that fictitious play -- the extreme version of FTRL -- can take doubly exponential time to reach a Nash equilibrium. This constitutes an exponentially stronger lower bound for the foundational learning algorithm in games.

Figures (5)

• Figure 1: Illustration of our results: $\texttt{FTRL}\xspace$ algorithms have the potential property (right), but can take exponential time to converge (left).
• Figure 2: A snake on a 4-dimensional hypercube (left) and a snake corresponding to a 3-player 4-action game (right).
• Figure 3: Probability mass shift between the two competing actions. Transitioning from $a_1(k-1)$ to $a_1(k)$ takes a long time when the initial gap is large. Eventually $a_2(k)$ will surpass $a_1(k)$, and this will trigger a shift in the other player's cumulative utility. The consistency of the argument requires bounding the time it takes for $a_2(k)$ to be played with high probability.
• Figure :
• Figure :

Theorems & Definitions (82)

• Theorem 1.1
• Theorem 1.2
• Theorem 1.3
• Proposition 2.1: Kalai05:EfficientShalev-Shwartz12:Online
• Example 2.2: $\texttt{MWU}\xspace$
• Example 2.3: Euclidean regularization
• Definition 2.4: Nash50:Equilibrium
• Definition 2.5
• Lemma 3.0
• Proposition 3.1