Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamics

TL;DR

This work addresses solving non-stationary mean field games in continuous state-action spaces by introducing Density-Enhanced Deep Average Fictitious Play (DEDA-FP). It extends fictitious play to continuous, time-varying MFGs by combining DRL-based best-response computation, supervised learning to represent the average policy, and a time-conditioned neural spline flow to model the evolving mean-field distribution. A convergence theorem provides an exploitability bound, ensuring convergence toward an $\epsilon$-Nash equilibrium under Lipschitz-type assumptions. Across Beach Bar, Linear-Quadratic, and 4-rooms exploration experiments, DEDA-FP demonstrates improved sampling efficiency, accurate density estimation, and robust policy learning in non-stationary settings, advancing practical applicability to real-world multi-agent systems.

Abstract

Mean field games (MFGs) have emerged as a powerful framework for modeling interactions in large-scale multi-agent systems. Despite recent advancements in reinforcement learning (RL) for MFGs, existing methods are typically limited to finite spaces or stationary models, hindering their applicability to real-world problems. This paper introduces a novel deep reinforcement learning (DRL) algorithm specifically designed for non-stationary continuous MFGs. The proposed approach builds upon a Fictitious Play (FP) methodology, leveraging DRL for best-response computation and supervised learning for average policy representation. Furthermore, it learns a representation of the time-dependent population distribution using a Conditional Normalizing Flow. To validate the effectiveness of our method, we evaluate it on three different examples of increasing complexity. By addressing critical limitations in scalability and density approximation, this work represents a significant advancement in applying DRL techniques to complex MFG problems, bringing the field closer to real-world multi-agent systems.

Figures (17)

• Figure 1: Overview of our DEDA-FP model. Our framework uses three main steps, built upon the Fictitious Play algorithm, to fully solve the MFG problem (details in Section \ref{['sec: approach']}): (1) computation of the best response using Deep RL algorithms; (2) learning a policy neural network to approximate the average policy over past policies; and (3) learning a Time-Conditioned Normalizing Flow to approximate the average distribution over past mean-field flows.
• Figure 2: Beach Bar Problem. a): exploitability of \ref{['algo:1']}, \ref{['algo:2']} and DEDA-FP; b) NE distribution DEDA-FP; c) NE distribution \ref{['algo:2']}; d) NE distribution \ref{['algo:1']}
• Figure 3: LQ Model DEDA-FP Left: mean field flow ${\boldsymbol{\mu}}^{\bar{{\boldsymbol{\pi}}}_{K}}$ at the last Fictitious Play (FP) iteration. Middle: policy $\bar{{\boldsymbol{\pi}}}_{K}$ at the last FP iteration. Right: comparison of exploitability between DEDA-FP and \ref{['algo:1']} and \ref{['algo:2']}.
• Figure 4: 4-rooms Exploration. Visualization of a large, finite population of 2000 agents and their velocity vectors during exploration in the 4-rooms environment. The agents' behavior is governed by the mean-field Nash equilibrium policy learned by the DEDA-FP solver, \ref{['alg:main-algo']}. This shows how well the mean-field approximation captures the behavior of a large-population system.
• Figure 5: 4-rooms Exploration - NE flow. The three plots represent the dynamics of the Nash Equilibrium mean field flow $\bar{{\boldsymbol{G}}}_K$ at time $t=6, 15, 20$, obtained by DEDA-FP. It can be seen how the population is spreading across the $4$ different rooms.

Theorems & Definitions (7)

• Definition 1
• Definition 2
• Theorem 1: Convergence to approximate Nash equilibrium
• Lemma 1
• proof
• Lemma 2
• proof