Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

$\widetilde{O}(T^{-1})$ Convergence to (Coarse) Correlated Equilibria in Full-Information General-Sum Markov Games

Weichao Mao, Haoran Qiu, Chen Wang, Hubertus Franke, Zbigniew Kalbarczyk, Tamer Başar

TL;DR

The paper tackles fast no-regret learning convergence to (coarse) correlated equilibria in full-information general-sum Markov games, expanding beyond normal-form results. It develops OFTRL-based algorithms—BM-OFTRL for CE with log-barrier regularization and stage-based OFTRL for CCE with negative entropy regularization—coupled with carefully designed value updates that leverage a Q-function oracle. The authors establish $\tilde{O}(T^{-1})$ convergence rates for CE and CCE, respectively, by bounding per-state swap regrets and using Regret by Variation in Utilities (RVU) properties, along with a certified policy construction. Numerical experiments corroborate the theoretical rates, indicating practical viability for decentralized multi-agent reinforcement learning in full-information settings. These results significantly narrow the gap between NFG and Markov game regimes in no-regret convergence to equilibrium concepts.

Abstract

No-regret learning has a long history of being closely connected to game theory. Recent works have devised uncoupled no-regret learning dynamics that, when adopted by all the players in normal-form games, converge to various equilibrium solutions at a near-optimal rate of $\widetilde{O}(T^{-1})$, a significant improvement over the $O(1/\sqrt{T})$ rate of classic no-regret learners. However, analogous convergence results are scarce in Markov games, a more generic setting that lays the foundation for multi-agent reinforcement learning. In this work, we close this gap by showing that the optimistic-follow-the-regularized-leader (OFTRL) algorithm, together with appropriate value update procedures, can find $\widetilde{O}(T^{-1})$-approximate (coarse) correlated equilibria in full-information general-sum Markov games within $T$ iterations. Numerical results are also included to corroborate our theoretical findings.

$\widetilde{O}(T^{-1})$ Convergence to (Coarse) Correlated Equilibria in Full-Information General-Sum Markov Games

TL;DR

The paper tackles fast no-regret learning convergence to (coarse) correlated equilibria in full-information general-sum Markov games, expanding beyond normal-form results. It develops OFTRL-based algorithms—BM-OFTRL for CE with log-barrier regularization and stage-based OFTRL for CCE with negative entropy regularization—coupled with carefully designed value updates that leverage a Q-function oracle. The authors establish convergence rates for CE and CCE, respectively, by bounding per-state swap regrets and using Regret by Variation in Utilities (RVU) properties, along with a certified policy construction. Numerical experiments corroborate the theoretical rates, indicating practical viability for decentralized multi-agent reinforcement learning in full-information settings. These results significantly narrow the gap between NFG and Markov game regimes in no-regret convergence to equilibrium concepts.

Abstract

No-regret learning has a long history of being closely connected to game theory. Recent works have devised uncoupled no-regret learning dynamics that, when adopted by all the players in normal-form games, converge to various equilibrium solutions at a near-optimal rate of , a significant improvement over the rate of classic no-regret learners. However, analogous convergence results are scarce in Markov games, a more generic setting that lays the foundation for multi-agent reinforcement learning. In this work, we close this gap by showing that the optimistic-follow-the-regularized-leader (OFTRL) algorithm, together with appropriate value update procedures, can find -approximate (coarse) correlated equilibria in full-information general-sum Markov games within iterations. Numerical results are also included to corroborate our theoretical findings.
Paper Structure (16 sections, 7 theorems, 53 equations, 2 figures, 3 tables, 4 algorithms)

This paper contains 16 sections, 7 theorems, 53 equations, 2 figures, 3 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Convergence to Correlated Equilibria
  4. Algorithm
  5. Analysis
  6. Convergence to Coarse Correlated Equilibria
  7. Algorithm
  8. Analysis
  9. Numerical Results
  10. Concluding Remarks
  11. Technical Lemmas
  12. Proofs for Section \ref{['sec:ce']}
  13. Supplementary Material for Section \ref{['sec:cce']}
  14. Policy Output Algorithm
  15. Proofs
  16. ...and 1 more sections

Key Result

Theorem 1

If Algorithm alg:oftrl is run on an $N$-player episodic Markov game for $T$ iterations with a learning rate $\eta = \frac{1}{256 NH \sqrt{HA_{\max}}}$, the output policy $\bar{\pi}$ satisfies:

Figures (2)

  • Figure 1: Convergence of $\operatorname{CCE/CE-Gap}(\bar{\pi})$
  • Figure 2: Convergence of $\operatorname{CCE/CE-Gap}(\bar{\pi}) \times T$

Theorems & Definitions (14)

  • Definition 1
  • Definition 2
  • Theorem 1
  • Lemma 1
  • Lemma 2
  • Theorem 2
  • Lemma 3
  • Lemma 4
  • proof
  • proof
  • ...and 4 more