Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Paths to Equilibrium in Games

Bora Yongacoglu, Gürdal Arslan, Lacra Pavel, Serdar Yüksel

TL;DR

The paper proves that every finite $n$-player normal-form game admits a satisficing path from any initial profile to a Nash equilibrium, by a constructive argument that iteratively enlarges the set of unsatisfied players and leverages boundary equilibria. A key, counterintuitive insight is that reward-deteriorating updates among unsatisfied players can drive convergence, offering a new design principle for MARL algorithms that mix conventional improvement steps with exploratory updates. The results extend the conceptual utility of satisficing dynamics beyond two-player or symmetric settings and discuss implications for Markov games, decentralized learning, and algorithmic complexity, while outlining open questions for epsilon-approximate equilibria and constrained strategy spaces. Overall, satisficing paths provide a robust alternative to best-response dynamics for achieving last-iterate convergence in general-sum games, informing practical MARL design.

Abstract

In multi-agent reinforcement learning (MARL) and game theory, agents repeatedly interact and revise their strategies as new data arrives, producing a sequence of strategy profiles. This paper studies sequences of strategies satisfying a pairwise constraint inspired by policy updating in reinforcement learning, where an agent who is best responding in one period does not switch its strategy in the next period. This constraint merely requires that optimizing agents do not switch strategies, but does not constrain the non-optimizing agents in any way, and thus allows for exploration. Sequences with this property are called satisficing paths, and arise naturally in many MARL algorithms. A fundamental question about strategic dynamics is such: for a given game and initial strategy profile, is it always possible to construct a satisficing path that terminates at an equilibrium? The resolution of this question has implications about the capabilities or limitations of a class of MARL algorithms. We answer this question in the affirmative for normal-form games. Our analysis reveals a counterintuitive insight that reward deteriorating strategic updates are key to driving play to equilibrium along a satisficing path.

Paths to Equilibrium in Games

TL;DR

The paper proves that every finite -player normal-form game admits a satisficing path from any initial profile to a Nash equilibrium, by a constructive argument that iteratively enlarges the set of unsatisfied players and leverages boundary equilibria. A key, counterintuitive insight is that reward-deteriorating updates among unsatisfied players can drive convergence, offering a new design principle for MARL algorithms that mix conventional improvement steps with exploratory updates. The results extend the conceptual utility of satisficing dynamics beyond two-player or symmetric settings and discuss implications for Markov games, decentralized learning, and algorithmic complexity, while outlining open questions for epsilon-approximate equilibria and constrained strategy spaces. Overall, satisficing paths provide a robust alternative to best-response dynamics for achieving last-iterate convergence in general-sum games, informing practical MARL design.

Abstract

In multi-agent reinforcement learning (MARL) and game theory, agents repeatedly interact and revise their strategies as new data arrives, producing a sequence of strategy profiles. This paper studies sequences of strategies satisfying a pairwise constraint inspired by policy updating in reinforcement learning, where an agent who is best responding in one period does not switch its strategy in the next period. This constraint merely requires that optimizing agents do not switch strategies, but does not constrain the non-optimizing agents in any way, and thus allows for exploration. Sequences with this property are called satisficing paths, and arise naturally in many MARL algorithms. A fundamental question about strategic dynamics is such: for a given game and initial strategy profile, is it always possible to construct a satisficing path that terminates at an equilibrium? The resolution of this question has implications about the capabilities or limitations of a class of MARL algorithms. We answer this question in the affirmative for normal-form games. Our analysis reveals a counterintuitive insight that reward deteriorating strategic updates are key to driving play to equilibrium along a satisficing path.
Paper Structure (14 sections, 2 theorems, 31 equations)

This paper contains 14 sections, 2 theorems, 31 equations.

Table of Contents

  1. Introduction
  2. Related Work.
  3. Normal-form games
  4. Satisficing Paths
  5. Existence of paths in normal-form games
  6. Proof of Theorem \ref{['theorem:main']}
  7. Algorithmic insights from the proof of Theorem 1
  8. Discussion
  9. Conclusion
  10. Properties of the auxiliary functions
  11. Proof of Lemma \ref{['lemma:equilibrium-on-boundary']}
  12. Markov games: model and connections to Theorem 1
  13. Normal-form game analysis.
  14. Markov game analysis.

Key Result

Theorem 1

Any finite normal-form game $\Gamma$ has the satisficing paths property.

Theorems & Definitions (12)

  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4
  • Definition 5
  • Theorem 1
  • Remark 1
  • proof
  • Lemma 1
  • proof
  • ...and 2 more