Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Simultaneous Best-Response Dynamics in Random Potential Games

Galit Ashkenazi-Golan, Domenico Mergoni Cecchelli, Edward Plumb

TL;DR

This work analyzes simultaneous best-response dynamics (SBRD) in random potential games, revealing a sharp division: in two-player settings with many actions, SBRD almost surely enters a two-cycle near a pair of Nash equilibria, while for three or more players it tends to converge quickly to a Nash equilibrium as the action set grows. The analysis combines a formal two-player convergence proof with extensive simulations for higher-player settings, and demonstrates robustness to payoff correlation, indicating similar behavior in near-potential games. Compared to softmax policy gradient (SPGD), SBRD achieves much faster convergence and favorable cumulative rewards, especially in large action spaces, albeit with marginally lower terminal payoffs in some cases. These findings support SBRD as a simple, scalable learning rule for decentralized multi-agent systems, while highlighting theoretical gaps for general n-player guarantees and potential extensions to partial observability.

Abstract

This paper examines the convergence behaviour of simultaneous best-response dynamics in random potential games. We provide a theoretical result showing that, for two-player games with sufficiently many actions, the dynamics converge quickly to a cycle of length two. This cycle lies within the intersection of the neighbourhoods of two distinct Nash equilibria. For three players or more, simulations show that the dynamics converge quickly to a Nash equilibrium with high probability. Furthermore, we show that all these results are robust, in the sense that they hold in non-potential games, provided the players' payoffs are sufficiently correlated. We also compare these dynamics to gradient-based learning methods in near-potential games with three players or more, and observe that simultaneous best-response dynamics converge to a Nash equilibrium of comparable payoff substantially faster.

Simultaneous Best-Response Dynamics in Random Potential Games

TL;DR

This work analyzes simultaneous best-response dynamics (SBRD) in random potential games, revealing a sharp division: in two-player settings with many actions, SBRD almost surely enters a two-cycle near a pair of Nash equilibria, while for three or more players it tends to converge quickly to a Nash equilibrium as the action set grows. The analysis combines a formal two-player convergence proof with extensive simulations for higher-player settings, and demonstrates robustness to payoff correlation, indicating similar behavior in near-potential games. Compared to softmax policy gradient (SPGD), SBRD achieves much faster convergence and favorable cumulative rewards, especially in large action spaces, albeit with marginally lower terminal payoffs in some cases. These findings support SBRD as a simple, scalable learning rule for decentralized multi-agent systems, while highlighting theoretical gaps for general n-player guarantees and potential extensions to partial observability.

Abstract

This paper examines the convergence behaviour of simultaneous best-response dynamics in random potential games. We provide a theoretical result showing that, for two-player games with sufficiently many actions, the dynamics converge quickly to a cycle of length two. This cycle lies within the intersection of the neighbourhoods of two distinct Nash equilibria. For three players or more, simulations show that the dynamics converge quickly to a Nash equilibrium with high probability. Furthermore, we show that all these results are robust, in the sense that they hold in non-potential games, provided the players' payoffs are sufficiently correlated. We also compare these dynamics to gradient-based learning methods in near-potential games with three players or more, and observe that simultaneous best-response dynamics converge to a Nash equilibrium of comparable payoff substantially faster.
Paper Structure (34 sections, 5 theorems, 27 equations, 8 figures)

This paper contains 34 sections, 5 theorems, 27 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Paper outline and summary of results
  3. Related work
  4. Learning dynamics in potential games
  5. Learning dynamics in games with random payoffs
  6. The setup
  7. Potential Games
  8. Random Potential Games
  9. The simultaneous best response algorithm
  10. Results
  11. Two Players
  12. Experimental Results
  13. Key findings.
  14. Technical details.
  15. Numerical Setup
  16. ...and 19 more sections

Key Result

Theorem 3.1

Let $\varepsilon\in (0,1)$, $F$ be a continuous real distribution, and $G$ be a two-player $m$-actions $F$-random potential game. If $m$ is large enough, then SBRD converges to a two-cycle in at most $\tfrac{\log\varepsilon}{\log(3/4)}$ steps with probability at least $1-\varepsilon$.

Figures (8)

  • Figure 1: SBRD in a two-player $50$-actions game. 10000 samples were drawn. Runtime: $22$ seconds.
  • Figure 2: SBRD in a three-player $50$-actions game. $1000$ samples were drawn. Runtime: $20$ minutes.
  • Figure 3: Comparison of SPGD and SBRD in a three-player $50$-actions game. $1000$ samples were drawn. Runtime: $80$ minutes.
  • Figure 4: SBRD in a two-player $500$-actions game. $1000$ samples were drawn. Runtime: $127$ seconds
  • Figure 5: SBRD in a three-player $100$-actions game. $1000$ samples were drawn. Runtime: $14$ minutes.
  • ...and 3 more figures

Theorems & Definitions (13)

  • Definition 2.1
  • Definition 2.2
  • Theorem 3.1
  • Lemma 3.1
  • Remark 3.2
  • Lemma 3.2
  • Lemma 3.2
  • Lemma 3.2
  • proof
  • proof
  • ...and 3 more