Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Provable Policy Gradient Methods for Average-Reward Markov Potential Games

Min Cheng, Ruida Zhou, P. R. Kumar, Chao Tian

TL;DR

The paper investigates infinite-horizon Markov potential games under the average reward criterion and develops policy-based MARL methods with provable guarantees. It proves that independent policy gradient, proximal-Q, and natural policy gradient converge globally to an ε-Nash equilibrium given a gradient/differential-value oracle, and establishes the smoothness of the average reward with respect to policies along with sensitivity bounds for differential value functions under ergodicity and spectral-gap conditions. When gradients are estimated from samples, it provides a single-trajectory gradient estimator with sample complexity $ ilde{O}(1/(π(a|s) δ))$ for ℓ2-error δ and a projected policy gradient with overall sample complexity $ ilde{O}(1/ε^5)$, along with an oracle-based complexity of $O(1/ε^2)$. Empirical results on synthetic AMPGs and a robot-navigation task validate the theory and demonstrate practical applicability of the single-trajectory gradient estimates in average-reward MARL.

Abstract

We study Markov potential games under the infinite horizon average reward criterion. Most previous studies have been for discounted rewards. We prove that both algorithms based on independent policy gradient and independent natural policy gradient converge globally to a Nash equilibrium for the average reward criterion. To set the stage for gradient-based methods, we first establish that the average reward is a smooth function of policies and provide sensitivity bounds for the differential value functions, under certain conditions on ergodicity and the second largest eigenvalue of the underlying Markov decision process (MDP). We prove that three algorithms, policy gradient, proximal-Q, and natural policy gradient (NPG), converge to an $ε$-Nash equilibrium with time complexity $O(\frac{1}{ε^2})$, given a gradient/differential Q function oracle. When policy gradients have to be estimated, we propose an algorithm with $\tilde{O}(\frac{1}{\min_{s,a}π(a|s)δ})$ sample complexity to achieve $δ$ approximation error w.r.t~the $\ell_2$ norm. Equipped with the estimator, we derive the first sample complexity analysis for a policy gradient ascent algorithm, featuring a sample complexity of $\tilde{O}(1/ε^5)$. Simulation studies are presented.

Provable Policy Gradient Methods for Average-Reward Markov Potential Games

TL;DR

The paper investigates infinite-horizon Markov potential games under the average reward criterion and develops policy-based MARL methods with provable guarantees. It proves that independent policy gradient, proximal-Q, and natural policy gradient converge globally to an ε-Nash equilibrium given a gradient/differential-value oracle, and establishes the smoothness of the average reward with respect to policies along with sensitivity bounds for differential value functions under ergodicity and spectral-gap conditions. When gradients are estimated from samples, it provides a single-trajectory gradient estimator with sample complexity for ℓ2-error δ and a projected policy gradient with overall sample complexity , along with an oracle-based complexity of . Empirical results on synthetic AMPGs and a robot-navigation task validate the theory and demonstrate practical applicability of the single-trajectory gradient estimates in average-reward MARL.

Abstract

We study Markov potential games under the infinite horizon average reward criterion. Most previous studies have been for discounted rewards. We prove that both algorithms based on independent policy gradient and independent natural policy gradient converge globally to a Nash equilibrium for the average reward criterion. To set the stage for gradient-based methods, we first establish that the average reward is a smooth function of policies and provide sensitivity bounds for the differential value functions, under certain conditions on ergodicity and the second largest eigenvalue of the underlying Markov decision process (MDP). We prove that three algorithms, policy gradient, proximal-Q, and natural policy gradient (NPG), converge to an -Nash equilibrium with time complexity , given a gradient/differential Q function oracle. When policy gradients have to be estimated, we propose an algorithm with sample complexity to achieve approximation error w.r.t~the norm. Equipped with the estimator, we derive the first sample complexity analysis for a policy gradient ascent algorithm, featuring a sample complexity of . Simulation studies are presented.
Paper Structure (31 sections, 41 theorems, 117 equations, 2 figures, 7 algorithms)

This paper contains 31 sections, 41 theorems, 117 equations, 2 figures, 7 algorithms.

Table of Contents

  1. INTRODUCTION
  2. Contributions
  3. Comparison with Previous Works
  4. Related Works
  5. Markov Potential Games
  6. Average Reward MDPs
  7. PRELIMINARIES
  8. Average Reward Markov Potential Games
  9. Value Functions and Their Properties
  10. Performance Metric
  11. POLICY GRADIENT ALGORITHM
  12. Policy Gradient Algorithm with Gradient Oracle
  13. Sample-Based Policy Gradient Estimate
  14. PROXIMAL-Q ALGORITHM
  15. NATURAL POLICY GRADIENT
  16. ...and 16 more sections

Key Result

Lemma 1

For any policies $\pi_j$, $\pi_j'$$\in\Pi_j$, and $\pi_{-j}\in\Pi_{-j}$, the difference between the average rewards for each agent $i$ is

Figures (2)

  • Figure 1: (a)(b)(c)(d) are results for the oracle setting. Since the Nash gap can be as low as $0.0$, we truncate the $\log(\text{Nash gap})$ at $-35$ from below. (d) depicts the change in $c(t)$ of the NPG algorithm for \ref{['fig:c']}. (e) and (f) present the results of \ref{['alg:2']}. The solid lines are the means of trajectories over seven random seeds and shaded regions are the standard deviations.
  • Figure 2: Training process of robot navigation task. The solid line is the mean of trajectories over three random seeds and shaded regions are the standard deviations.

Theorems & Definitions (68)

  • Definition 1: Average reward Markov potential games
  • Definition 2: Nash and $\epsilon$-Nash equilibrium
  • Lemma 1: Performance difference lemma
  • Lemma 2: Partial derivative
  • Definition 3: Li
  • Definition 4
  • Lemma 3
  • Definition 5
  • Lemma 4: Smoothness of $\rho$ and $\Phi$
  • Definition 6: Distribution mismatch coefficient
  • ...and 58 more