Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Independent Learning in Performative Markov Potential Games

Rilind Sahitaj, Paulius Sasnauskas, Yiğit Yalın, Debmalya Mandal, Goran Radanović

TL;DR

This work extends multi-agent reinforcement learning to performative environments by embedding performative effects into Markov Potential Games and introducing performatively stable equilibrium (PSE). It develops convergence guarantees for independent policy gradient methods (IPGA and INPG) to approximate PSEs, including best-iterate and last-iterate results, and shows how performativity introduces an additive cost term that vanishes as the effects diminish. A finite-time last-iterate result is provided for a special occupancy-measure optimization subclass via a repeated retraining scheme, complemented by extensive experiments demonstrating robustness of natural policy gradients under performativity. The framework unifies a theoretical treatment of performativity with MPGs and offers practical guidance for stable, decentralized learning in environments that respond to deployed policies.

Abstract

Performative Reinforcement Learning (PRL) refers to a scenario in which the deployed policy changes the reward and transition dynamics of the underlying environment. In this work, we study multi-agent PRL by incorporating performative effects into Markov Potential Games (MPGs). We introduce the notion of a performatively stable equilibrium (PSE) and show that it always exists under a reasonable sensitivity assumption. We then provide convergence results for state-of-the-art algorithms used to solve MPGs. Specifically, we show that independent policy gradient ascent (IPGA) and independent natural policy gradient (INPG) converge to an approximate PSE in the best-iterate sense, with an additional term that accounts for the performative effects. Furthermore, we show that INPG asymptotically converges to a PSE in the last-iterate sense. As the performative effects vanish, we recover the convergence rates from prior work. For a special case of our game, we provide finite-time last-iterate convergence results for a repeated retraining approach, in which agents independently optimize a surrogate objective. We conduct extensive experiments to validate our theoretical findings.

Independent Learning in Performative Markov Potential Games

TL;DR

This work extends multi-agent reinforcement learning to performative environments by embedding performative effects into Markov Potential Games and introducing performatively stable equilibrium (PSE). It develops convergence guarantees for independent policy gradient methods (IPGA and INPG) to approximate PSEs, including best-iterate and last-iterate results, and shows how performativity introduces an additive cost term that vanishes as the effects diminish. A finite-time last-iterate result is provided for a special occupancy-measure optimization subclass via a repeated retraining scheme, complemented by extensive experiments demonstrating robustness of natural policy gradients under performativity. The framework unifies a theoretical treatment of performativity with MPGs and offers practical guidance for stable, decentralized learning in environments that respond to deployed policies.

Abstract

Performative Reinforcement Learning (PRL) refers to a scenario in which the deployed policy changes the reward and transition dynamics of the underlying environment. In this work, we study multi-agent PRL by incorporating performative effects into Markov Potential Games (MPGs). We introduce the notion of a performatively stable equilibrium (PSE) and show that it always exists under a reasonable sensitivity assumption. We then provide convergence results for state-of-the-art algorithms used to solve MPGs. Specifically, we show that independent policy gradient ascent (IPGA) and independent natural policy gradient (INPG) converge to an approximate PSE in the best-iterate sense, with an additional term that accounts for the performative effects. Furthermore, we show that INPG asymptotically converges to a PSE in the last-iterate sense. As the performative effects vanish, we recover the convergence rates from prior work. For a special case of our game, we provide finite-time last-iterate convergence results for a repeated retraining approach, in which agents independently optimize a surrogate objective. We conduct extensive experiments to validate our theoretical findings.
Paper Structure (23 sections, 9 theorems, 35 equations, 2 figures, 1 table)

This paper contains 23 sections, 9 theorems, 35 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Markov Potential Games.
  4. Performative Prediction.
  5. Performative Reinforcement Learning.
  6. Formal Setting
  7. Performative Markov Game
  8. Performative Markov Potential Games
  9. Solution Concepts
  10. Learning Objective
  11. Characterization of Solution Concepts
  12. Independent Gradient methods
  13. Independent Gradient Methods with Oracle
  14. Independent Policy Gradient Ascent
  15. Independent Natural Policy Gradients
  16. ...and 8 more sections

Key Result

Lemma 1

For any state distribution $\rho \in \Delta(S)$, there exists a policy $\pi^* \in \Pi$ such that for all $i \in \mathcal{N}$, $\pi_i \in \Pi_i$.

Figures (2)

  • Figure 1: Comparison of IPGA-L and IPGA-D, showing the distance from the current policy to the average of the last 10 in that run: $\frac{1}{n} \sum_i^n \left\| \pi_i^t - \pi_i^\text{last} \right\|$, $\gamma = 0.99$. Left: IPGA-L $\eta = 0.00001$, IPGA-D $\eta = 0.0001$. Right: $\eta = 0.0003$.
  • Figure 2: Comparison of IPGA-D, INPG (unreg.), INPG (reg.), showing the distance from the current policy to the average of the last 10 in that run: $\frac{1}{N} \sum_i^N \left\| \pi_i^t - \pi_i^\text{last} \right\|$, $\gamma = 0.99$. Left: $\alpha = 0.01$, $\eta = 0.0001$. Right: $\omega = 0.03$, $\eta = 0.0006$.

Theorems & Definitions (12)

  • Definition 1: $\epsilon$-NE
  • Definition 2: $\epsilon$-PSE
  • Lemma 1
  • Theorem 1
  • Lemma 2
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Theorem 5
  • Theorem 6
  • ...and 2 more