Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Equilibrium Selection for Multi-agent Reinforcement Learning: A Unified Framework

Runyu Zhang, Gioele Zardini, Asuman Ozdaglar, Jeff Shamma, Na Li

TL;DR

This work addresses the challenge of selecting high-quality equilibria in finite-horizon stochastic games (SGs) by exporting normal-form equilibrium-selection ideas into the SG setting. The authors propose a modular actor–critic framework where a critic learns stagewise $Q_{i,h}^\pi(s,a)$ values and an actor applies a chosen normal-form learning rule to these $Q$-values at each state and stage, effectively reducing SG equilibrium selection to normal-form dynamics. They prove that, under standard ergodicity and resistance assumptions, the stochastically stable policies in the SG inherit the equilibrium-selection properties of the embedded normal-form rule: in Markov potential games with log-linear learning the selected policies maximize the potential, while in general-sum SGs with suitable learning rules the selected policies are Pareto-optimal MPEs. The framework also includes a fully sample-based variant and a two-stage stag-hunt numerical study demonstrating how different rules steer the system toward different high-quality equilibria. Overall, the paper provides a principled, plug-and-play approach to steer MARL toward socially desirable equilibria with theoretical guarantees and practical algorithms.

Abstract

While multi-agent reinforcement learning (MARL) has produced numerous algorithms that converge to Nash or related equilibria, such equilibria are often non-unique and can exhibit widely varying efficiency. This raises a fundamental question: how can one design learning dynamics that not only converge to equilibrium but also select equilibria with desirable performance, such as high social welfare? In contrast to the MARL literature, equilibrium selection has been extensively studied in normal-form games, where decentralized dynamics are known to converge to potential-maximizing or Pareto-optimal Nash equilibria (NEs). Motivated by these results, we study equilibrium selection in finite-horizon stochastic games. We propose a unified actor-critic framework in which a critic learns state-action value functions, and an actor applies a classical equilibrium-selection rule state-wise, treating learned values as stage-game payoffs. We show that, under standard stochastic stability assumptions, the stochastically stable policies of the resulting dynamics inherit the equilibrium selection properties of the underlying normal-form learning rule. As consequences, we obtain potential-maximizing policies in Markov potential games and Pareto-optimal (Markov perfect) equilibria in general-sum stochastic games, together with sample-based implementation of the framework.

Equilibrium Selection for Multi-agent Reinforcement Learning: A Unified Framework

TL;DR

This work addresses the challenge of selecting high-quality equilibria in finite-horizon stochastic games (SGs) by exporting normal-form equilibrium-selection ideas into the SG setting. The authors propose a modular actor–critic framework where a critic learns stagewise values and an actor applies a chosen normal-form learning rule to these -values at each state and stage, effectively reducing SG equilibrium selection to normal-form dynamics. They prove that, under standard ergodicity and resistance assumptions, the stochastically stable policies in the SG inherit the equilibrium-selection properties of the embedded normal-form rule: in Markov potential games with log-linear learning the selected policies maximize the potential, while in general-sum SGs with suitable learning rules the selected policies are Pareto-optimal MPEs. The framework also includes a fully sample-based variant and a two-stage stag-hunt numerical study demonstrating how different rules steer the system toward different high-quality equilibria. Overall, the paper provides a principled, plug-and-play approach to steer MARL toward socially desirable equilibria with theoretical guarantees and practical algorithms.

Abstract

While multi-agent reinforcement learning (MARL) has produced numerous algorithms that converge to Nash or related equilibria, such equilibria are often non-unique and can exhibit widely varying efficiency. This raises a fundamental question: how can one design learning dynamics that not only converge to equilibrium but also select equilibria with desirable performance, such as high social welfare? In contrast to the MARL literature, equilibrium selection has been extensively studied in normal-form games, where decentralized dynamics are known to converge to potential-maximizing or Pareto-optimal Nash equilibria (NEs). Motivated by these results, we study equilibrium selection in finite-horizon stochastic games. We propose a unified actor-critic framework in which a critic learns state-action value functions, and an actor applies a classical equilibrium-selection rule state-wise, treating learned values as stage-game payoffs. We show that, under standard stochastic stability assumptions, the stochastically stable policies of the resulting dynamics inherit the equilibrium selection properties of the underlying normal-form learning rule. As consequences, we obtain potential-maximizing policies in Markov potential games and Pareto-optimal (Markov perfect) equilibria in general-sum stochastic games, together with sample-based implementation of the framework.
Paper Structure (20 sections, 18 theorems, 59 equations, 4 figures, 1 table, 2 algorithms)

This paper contains 20 sections, 18 theorems, 59 equations, 4 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Settings and Preliminaries
  3. The stochastic game model
  4. Informal statement of the main result - equilibrium selection for SGs
  5. Equilibrium selection for normal-form games
  6. Equilibrium selection for stochastic games: a unified framework
  7. Fully sample-based algorithm
  8. Proof sketches for Theorem \ref{['theorem:stochastic-stable-global-optimality']}
  9. Key technical challenge: iteration-varying ${\{Q_{i,h}^{(t)}\}_{i=1}^n}$
  10. To gain intuition: consider two-stage game $H=2$
  11. Extension to $H > 2$, general-sum games and other learning rules
  12. Numerical case study: a two-stage stag-hunt
  13. Conclusions
  14. Proof of Theorem \ref{['theorem:stochastic-stable-global-optimality']}
  15. Proof of Corollary \ref{['coro:SG-log-linear-learning']}, \ref{['coro:SG-pareto-optimal']}, \ref{['coro:SG-pareto-optimal-NE']}
  16. ...and 5 more sections

Key Result

Theorem 2.1

An action $a^\star\in\mathcal{A}$ is the SSE of a normal-form game ${\{r_i\}_{i=1}^n}$ and learning rule ${K^\epsilon}$ if and only if there exists a hidden variable $\xi^\star\in\mathcal{E}$ such that $a^\star, \xi^\star$ minimizes the stochastic potential, i.e., $\gamma(a^\star, \xi^\star) = \min_

Figures (4)

  • Figure 1: Game schematic for the two-stage stag-hunt game (states, transitions, payoffs).
  • Figure 1: Game schematic (states, transitions, payoffs).
  • Figure 2: Numerical result of running Algorithm \ref{['alg:unified-framework']} with log-linear learning (Left) and the learning rule in marden_achieving_2012 (Right) The plot demonstrates the evolution of $\pi^{(t)}(a|s)$ (estimated using empirical averaging), where $s$ is the initial state and the yellow line plots the curve for $a = (1,1)$ and the blue line $a=(0,0)$. The shaded areas are the $60\%$ confidence interval, which are calculated by 100 runs.
  • Figure 2: (Left) Illustration of 2-stage the stochastic game. (Right) Numerical result of running Algorithm \ref{['alg:unified-framework']} with log-linear learning rule (with $\epsilon = 10^{-5}$). The plot demonstrates the evolution of $\pi^{(t)}(a|s)$ (estimated using empirical averaging), where $s$ is the initial state and the yellow line plots the curve for $a = (1,1)$ and the blue line $a=(0,0)$. The shaded areas are the $60\%$ confidence interval calculated by 100 runs.

Theorems & Definitions (43)

  • Definition 1: Markov perfect (Nash) equilibrium (MPE)
  • Definition 2: Pareto optimal policy and Pareto optimal MPE
  • Definition 3: Markov potential game (MPG)
  • Definition 4: Potential-maximizing policy
  • Example 1: Log-linear learning blume_statistical_1993marden_revisiting_2012
  • Example 2: marden_achieving_2012
  • Definition 5: Stationary distribution ${\pi^\epsilon}$
  • Definition 6: Stochastically stable equilibrium (SSE) foster1990stochasticyoung_evolution_1993
  • Definition 7: Resistance
  • Definition 8: Stochastic potential
  • ...and 33 more