Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarial Inverse Reinforcement Learning for Mean Field Games

Yang Chen, Libo Zhang, Jiamou Liu, Michael Witbrock

TL;DR

This work tackles inverse reinforcement learning in mean-field games to model and design large-scale multi-agent systems under bounded rationality. It introduces Mean-Field Adversarial IRL (MF-AIRL), built on entropy-regularised MFNE (ERMFNE) and adversarial MaxEnt IRL, to recover reward signals from imperfect demonstrations. The authors derive a tractable extension of MaxEnt IRL to MFGs by substituting the empirical mean field and frame MF-AIRL as an adversarial learning problem with a discriminator and adaptive samplers, including a reward-shaping invariance via a potential function. Empirical results on five large-scale tasks show MF-AIRL outperforms centralised and decentralised baselines in reward recovery and robustness to changes in dynamics, demonstrating practical viability for environment design and behavior prediction in large MAS.

Abstract

Mean field games (MFGs) provide a mathematically tractable framework for modelling large-scale multi-agent systems by leveraging mean field theory to simplify interactions among agents. It enables applying inverse reinforcement learning (IRL) to predict behaviours of large populations by recovering reward signals from demonstrated behaviours. However, existing IRL methods for MFGs are powerless to reason about uncertainties in demonstrated behaviours of individual agents. This paper proposes a novel framework, Mean-Field Adversarial IRL (MF-AIRL), which is capable of tackling uncertainties in demonstrations. We build MF-AIRL upon maximum entropy IRL and a new equilibrium concept. We evaluate our approach on simulated tasks with imperfect demonstrations. Experimental results demonstrate the superiority of MF-AIRL over existing methods in reward recovery.

Adversarial Inverse Reinforcement Learning for Mean Field Games

TL;DR

This work tackles inverse reinforcement learning in mean-field games to model and design large-scale multi-agent systems under bounded rationality. It introduces Mean-Field Adversarial IRL (MF-AIRL), built on entropy-regularised MFNE (ERMFNE) and adversarial MaxEnt IRL, to recover reward signals from imperfect demonstrations. The authors derive a tractable extension of MaxEnt IRL to MFGs by substituting the empirical mean field and frame MF-AIRL as an adversarial learning problem with a discriminator and adaptive samplers, including a reward-shaping invariance via a potential function. Empirical results on five large-scale tasks show MF-AIRL outperforms centralised and decentralised baselines in reward recovery and robustness to changes in dynamics, demonstrating practical viability for environment design and behavior prediction in large MAS.

Abstract

Mean field games (MFGs) provide a mathematically tractable framework for modelling large-scale multi-agent systems by leveraging mean field theory to simplify interactions among agents. It enables applying inverse reinforcement learning (IRL) to predict behaviours of large populations by recovering reward signals from demonstrated behaviours. However, existing IRL methods for MFGs are powerless to reason about uncertainties in demonstrated behaviours of individual agents. This paper proposes a novel framework, Mean-Field Adversarial IRL (MF-AIRL), which is capable of tackling uncertainties in demonstrations. We build MF-AIRL upon maximum entropy IRL and a new equilibrium concept. We evaluate our approach on simulated tasks with imperfect demonstrations. Experimental results demonstrate the superiority of MF-AIRL over existing methods in reward recovery.

Paper Structure

This paper contains 44 sections, 4 theorems, 34 equations, 2 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Mean Field Games
  4. Mean Field Nash Equilibrium
  5. Maximum Entropy IRL
  6. IRL for MFGs
  7. Entropy-Regularised MFNE
  8. A New Equilibrium Concept for MFGs
  9. Trajectory Distributions under ERMFNE
  10. Extending MaxEnt IRL to MFGs
  11. Mean Field Adversarial IRL
  12. Reward Shaping in MFGs
  13. Related Work
  14. Experiments
  15. Experimental Setup
  16. ...and 29 more sections

Key Result

Theorem 1

Let $(\tilde{{\bm{\mu}}}^\star, \tilde{{\bm{\pi}}}^\star)$ be the ERMFNE for an MFG $({{\mathcal{S}}}, {{\mathcal{A}}}, P,$$\mu_0, r)$, and $D_{\mathrm{KL}}$ denote the Kullback-Leibler (KL) divergence. Then, $(\tilde{{\bm{\mu}}}^\star, \tilde{{\bm{\pi}}}^\star)$ is the optimal solution to the follo

Figures (2)

  • Figure 1: Illustration for the trajectory distribution induced by ERMFNE. A group of agents (dots) drift in a grid world over three time steps. Each grid represents a specific state. The distribution of agents represents the mean field. A darker grey background denotes a higher reward associated with a state. Two individual agents marked with "$\star$" and "$\bm{+}$" start at the same state. The solid arrows denote the trajectory of $\star$, which can be observed with an exponentially higher probability than the trajectory of $\bm{+}$ depicted using dashed arrows. Note that the rewards of states change over time because of the evolution of mean fields.
  • Figure 2: Results for numerical tasks. The line and shade are the median and variance over 10 independent runs.

Theorems & Definitions (6)

  • Definition 1
  • Definition 2
  • Theorem 1
  • Theorem 2
  • Lemma A.1: cui2021approximately
  • Theorem 3