Learning in Stackelberg Mean Field Games: A Non-Asymptotic Analysis

TL;DR

The paper tackles learning in Stackelberg mean field games by introducing AC-SMFG, a single-loop actor-critic algorithm that simultaneously updates the leader policy, a representative follower policy, and the mean field using continuously generated Markovian samples. It provides the first non-asymptotic convergence guarantees for SMFGs, achieving a rate of $ ilde{O}(k^{-1/2})$ to a stationary point under a gradient-alignment condition that relaxes leader-follower independence. The method relies on multi-time-scale updates and entropy-regularized objectives to obtain a contraction in the lower-level mean field response, with two samples drawn per iteration yielding a favorable sample complexity. Empirical results in economics-inspired environments demonstrate faster convergence and higher-quality equilibria than existing multi-agent and MFG baselines, and the approach accommodates function approximation, suggesting broad practical impact for hierarchical decision-making in large populations.

Abstract

We study policy optimization in Stackelberg mean field games (MFGs), a hierarchical framework for modeling the strategic interaction between a single leader and an infinitely large population of homogeneous followers. The objective can be formulated as a structured bi-level optimization problem, in which the leader needs to learn a policy maximizing its reward, anticipating the response of the followers. Existing methods for solving these (and related) problems often rely on restrictive independence assumptions between the leader's and followers' objectives, use samples inefficiently due to nested-loop algorithm structure, and lack finite-time convergence guarantees. To address these limitations, we propose AC-SMFG, a single-loop actor-critic algorithm that operates on continuously generated Markovian samples. The algorithm alternates between (semi-)gradient updates for the leader, a representative follower, and the mean field, and is simple to implement in practice. We establish the finite-time and finite-sample convergence of the algorithm to a stationary point of the Stackelberg objective. To our knowledge, this is the first Stackelberg MFG algorithm with non-asymptotic convergence guarantees. Our key assumption is a "gradient alignment" condition, which requires that the full policy gradient of the leader can be approximated by a partial component of it, relaxing the existing leader-follower independence assumption. Simulation results in a range of well-established economics environments demonstrate that AC-SMFG outperforms existing multi-agent and MFG learning baselines in policy quality and convergence speed.

Figures (8)

• Figure 1: Theoretical convergence rates. The proposed algorithm is the first to have non-asymptotic convergence guarantees (black). Existing algorithms in similar settings are only known to converge asymptotically (red). Convergence rate of a single-loop algorithm for bi-level optimization hong2023two under a non-convex upper-level objective and lower-level strong convexity (blue).
• Figure 2: Convergence across environments, demonstrating bootstrapped mean and 5% confidence interval across 30 runs for the leader reward (left) and follower rewards (right).
• Figure 3: Equilibria Analysis
• Figure 4: Function approximation with continuous states/actions in Equilibrium Pricing (with $i=0$)
• Figure 5: Beach Bar Environment. The blue stars indicates the fixed bar position, and the golden star shows the leader can add a new bar anywhere within the existing state.

Theorems & Definitions (17)

• Theorem 1
• Remark 1
• Remark 2
• Remark 3: Continuous State and Action Spaces & Function approximation
• Lemma 1
• Lemma 2
• Lemma 3
• Lemma 4
• Theorem 2
• Proposition 1: Leader Convergence