Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Independent and Decentralized Learning in Markov Potential Games

Chinmay Maheshwari, Manxi Wu, Druv Pai, Shankar Sastry

TL;DR

The paper addresses decentralized multi-agent reinforcement learning in finite-state Markov Potential Games (MPGs) where agents lack knowledge of game parameters or coordination. It proposes a two-timescale, asynchronous actor-critic framework in which Q-function estimates are updated faster than policies, and policy updates incorporate optimal one-stage deviations based on the estimated Q-function. Using two-timescale stochastic approximation and a Lyapunov analysis of the potential function $\Phi$, it characterizes the convergent set as the smallest super-level set $\Pi^*_\epsilon$ that contains the $\epsilon$-stationary Nash equilibria, with corollaries describing convergence to $\textsf{NE}(\epsilon+h_\epsilon)$ under regularity. Numerical experiments validate convergence to approximate NE in a decentralized Markov routing game, showing that smaller exploration improves the Nash gap, highlighting the practical viability of decentralized learning in MPGs.

Abstract

We study a multi-agent reinforcement learning dynamics, and analyze its asymptotic behavior in infinite-horizon discounted Markov potential games. We focus on the independent and decentralized setting, where players do not know the game parameters, and cannot communicate or coordinate. In each stage, players update their estimate of Q-function that evaluates their total contingent payoff based on the realized one-stage reward in an asynchronous manner. Then, players independently update their policies by incorporating an optimal one-stage deviation strategy based on the estimated Q-function. Inspired by the actor-critic algorithm in single-agent reinforcement learning, a key feature of our learning dynamics is that agents update their Q-function estimates at a faster timescale than the policies. Leveraging tools from two-timescale asynchronous stochastic approximation theory, we characterize the convergent set of learning dynamics.

Independent and Decentralized Learning in Markov Potential Games

TL;DR

The paper addresses decentralized multi-agent reinforcement learning in finite-state Markov Potential Games (MPGs) where agents lack knowledge of game parameters or coordination. It proposes a two-timescale, asynchronous actor-critic framework in which Q-function estimates are updated faster than policies, and policy updates incorporate optimal one-stage deviations based on the estimated Q-function. Using two-timescale stochastic approximation and a Lyapunov analysis of the potential function , it characterizes the convergent set as the smallest super-level set that contains the -stationary Nash equilibria, with corollaries describing convergence to under regularity. Numerical experiments validate convergence to approximate NE in a decentralized Markov routing game, showing that smaller exploration improves the Nash gap, highlighting the practical viability of decentralized learning in MPGs.

Abstract

We study a multi-agent reinforcement learning dynamics, and analyze its asymptotic behavior in infinite-horizon discounted Markov potential games. We focus on the independent and decentralized setting, where players do not know the game parameters, and cannot communicate or coordinate. In each stage, players update their estimate of Q-function that evaluates their total contingent payoff based on the realized one-stage reward in an asynchronous manner. Then, players independently update their policies by incorporating an optimal one-stage deviation strategy based on the estimated Q-function. Inspired by the actor-critic algorithm in single-agent reinforcement learning, a key feature of our learning dynamics is that agents update their Q-function estimates at a faster timescale than the policies. Leveraging tools from two-timescale asynchronous stochastic approximation theory, we characterize the convergent set of learning dynamics.
Paper Structure (14 sections, 10 theorems, 87 equations, 1 figure, 1 algorithm)

This paper contains 14 sections, 10 theorems, 87 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Additional related Works
  3. Model
  4. Independent and Decentralized Learning Dynamics
  5. Learning Dynamics
  6. Convergence Analysis
  7. Numerical experiments
  8. Conclusions
  9. Review of Two-timescale asynchronous stochastic approximation
  10. Remaining Proofs
  11. Proof of Lemma \ref{['lemma:d_fast']}
  12. Proof of Lemma \ref{['lem: TechnicalMain']}
  13. Proof of Corollary \ref{['cor: RefineConvergenceNew']}
  14. Auxiliary Lemma

Key Result

Theorem 3.3

Under Assumptions as:basic and as:stepsize, for every $\epsilon > 0$, the policy sequence $\{\pi^t\}_{t=0}^{\infty}$ induced by Algorithm alg:independent_decentralized converges to the set with probability 1 given that where and $A_{\zeta}$ is defined as in Assumption as:stepsize (iii). Moreover, for any $\epsilon,\epsilon'$ such that $0\leq \epsilon\leq \epsilon'$, $\Pi^\ast_{\epsilon}\subsete

Figures (1)

  • Figure 1: Variation of Nash approximation gap during $10^4$ steps of Algorithm \ref{['alg:independent_decentralized']}. The first (resp. second) figure shows the variation with exploration probability $\theta_i = 0.1$ (resp. $\theta_i = 0.2$), for every $i\in I.$ In each of the figures the four curves correspond to four players. Each curve represents the mean value of the quantity over $5$ trials, and we give error margins of $\pm 1$ standard deviation.

Theorems & Definitions (17)

  • Definition 2.1: Markov potential games leonardos2021global
  • Definition 2.2: Stationary Nash equilibrium policy
  • Definition 2.3: $\epsilon$-Stationary Nash equilibrium policy
  • Theorem 3.3
  • Corollary 3.5
  • Lemma 3.6
  • Lemma 3.7
  • Lemma 3.8
  • Lemma 3.9
  • proof
  • ...and 7 more