Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convex Markov Games and Beyond: New Proof of Existence, Characterization and Learning Algorithms for Nash Equilibria

Anas Barakat, Ioannis Panageas, Antonios Varvitsiotis

TL;DR

This work proves that in GUMGs, Nash equilibria coincide with the fixed points of projected pseudo-gradient dynamics (i.e., first-order stationary points), enabled by a novel agent-wise gradient domination property, and establishes a policy gradient theorem for GUMGs and designs a model-free policy gradient algorithm.

Abstract

Convex Markov Games (cMGs) were recently introduced as a broad class of multi-agent learning problems that generalize Markov games to settings where strategic agents optimize general utilities beyond additive rewards. While cMGs expand the modeling frontier, their theoretical foundations, particularly the structure of Nash equilibria (NE) and guarantees for learning algorithms, are not yet well understood. In this work, we address these gaps for an extension of cMGs, which we term General Utility Markov Games (GUMGs), capturing new applications requiring coupling between agents' occupancy measures. We prove that in GUMGs, Nash equilibria coincide with the fixed points of projected pseudo-gradient dynamics (i.e., first-order stationary points), enabled by a novel agent-wise gradient domination property. This insight also yields a simple proof of NE existence using Brouwer's fixed-point theorem. We further show the existence of Markov perfect equilibria. Building on this characterization, we establish a policy gradient theorem for GUMGs and design a model-free policy gradient algorithm. For potential GUMGs, we establish iteration complexity guarantees for computing approximate-NE under exact gradients and provide sample complexity bounds in both the generative model and on-policy settings. Our results extend beyond prior work restricted to zero-sum cMGs, providing the first theoretical analysis of common-interest cMGs.

Convex Markov Games and Beyond: New Proof of Existence, Characterization and Learning Algorithms for Nash Equilibria

TL;DR

This work proves that in GUMGs, Nash equilibria coincide with the fixed points of projected pseudo-gradient dynamics (i.e., first-order stationary points), enabled by a novel agent-wise gradient domination property, and establishes a policy gradient theorem for GUMGs and designs a model-free policy gradient algorithm.

Abstract

Convex Markov Games (cMGs) were recently introduced as a broad class of multi-agent learning problems that generalize Markov games to settings where strategic agents optimize general utilities beyond additive rewards. While cMGs expand the modeling frontier, their theoretical foundations, particularly the structure of Nash equilibria (NE) and guarantees for learning algorithms, are not yet well understood. In this work, we address these gaps for an extension of cMGs, which we term General Utility Markov Games (GUMGs), capturing new applications requiring coupling between agents' occupancy measures. We prove that in GUMGs, Nash equilibria coincide with the fixed points of projected pseudo-gradient dynamics (i.e., first-order stationary points), enabled by a novel agent-wise gradient domination property. This insight also yields a simple proof of NE existence using Brouwer's fixed-point theorem. We further show the existence of Markov perfect equilibria. Building on this characterization, we establish a policy gradient theorem for GUMGs and design a model-free policy gradient algorithm. For potential GUMGs, we establish iteration complexity guarantees for computing approximate-NE under exact gradients and provide sample complexity bounds in both the generative model and on-policy settings. Our results extend beyond prior work restricted to zero-sum cMGs, providing the first theoretical analysis of common-interest cMGs.
Paper Structure (42 sections, 26 theorems, 160 equations, 1 figure, 3 tables, 1 algorithm)

This paper contains 42 sections, 26 theorems, 160 equations, 1 figure, 3 tables, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. Our Contributions
  3. Technical Overview
  4. Related Work
  5. FROM CLASSICAL TO GENERAL UTILITY MARKOV GAMES
  6. Markov Games and Notation
  7. Convex Markov Games
  8. General Utility Markov Games
  9. Examples
  10. EXISTENCE AND STRUCTURE OF NASH POLICIES IN GUMGs
  11. POLICY GRADIENT ALGORITHM FOR GUMGs
  12. LEARNING NASH EQUILIBRIA IN POTENTIAL GUMGs
  13. Exact Setting Iteration Complexity
  14. Generative Model Sample Complexity
  15. On-policy Learning Sample Complexity
  16. ...and 27 more sections

Key Result

Proposition 1

Let Assumptions as:concavity and as:exploration hold. For all $i \in \mathcal{N}$, all policies $\pi = (\pi_i, \pi_{-i}) \in \Pi$, $\pi_i' \in \Pi_i$, we have for any distribution $\mu, \rho \in \Delta({{\mathcal{S}}})$ satisfying Assumption as:exploration, where the minimax distribution mismatch coefficient: $C_{\mathcal{G}} := \max_{i \in \mathcal{N}} \max_{\pi \in \Pi} \min_{\pi_i^{\star} \in

Figures (1)

  • Figure 1: From single-agent RL to General Utility Markov Games (GUMG). MDP: Markov Decision Processes (single-agent RL), MG: Markov Games (MARL), cMG: convex Markov Games.

Theorems & Definitions (52)

  • Definition 1: General Utility Markov Game
  • Remark 1
  • Proposition 1: Agentwise Gradient Domination
  • Remark 2
  • Proposition 2: First-order Stationary policies are Nash policies
  • Theorem 1
  • Corollary 1
  • proof
  • Proposition 3: Multi-Agent PG Theorem
  • Definition 2: Potential GUMG
  • ...and 42 more