Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Distributed Equilibria in Linear-Quadratic Stochastic Differential Games: An $α$-Potential Approach

Philipp Plank, Yufei Zhang

TL;DR

It is proved that independent projected PG algorithms converge linearly to an approximate equilibrium, with suboptimality proportional to the degree of asymmetry, across both symmetric and asymmetric interaction networks.

Abstract

We analyze independent policy-gradient (PG) learning in $N$-player linear-quadratic (LQ) stochastic differential games. Each player employs a distributed policy that depends only on its own state and updates the policy independently using the gradient of its own objective. We establish global linear convergence of these methods to an equilibrium by showing that the LQ game admits an $α$-potential structure, with $α$ determined by the degree of pairwise interaction asymmetry. For pairwise-symmetric interactions, we construct an affine distributed equilibrium by minimizing the potential function and show that independent PG methods converge globally to this equilibrium, with complexity scaling linearly in the population size and logarithmically in the desired accuracy. For asymmetric interactions, we prove that independent projected PG algorithms converge linearly to an approximate equilibrium, with suboptimality proportional to the degree of asymmetry. Numerical experiments confirm the theoretical results across both symmetric and asymmetric interaction networks.

Learning Distributed Equilibria in Linear-Quadratic Stochastic Differential Games: An $α$-Potential Approach

TL;DR

It is proved that independent projected PG algorithms converge linearly to an approximate equilibrium, with suboptimality proportional to the degree of asymmetry, across both symmetric and asymmetric interaction networks.

Abstract

We analyze independent policy-gradient (PG) learning in -player linear-quadratic (LQ) stochastic differential games. Each player employs a distributed policy that depends only on its own state and updates the policy independently using the gradient of its own objective. We establish global linear convergence of these methods to an equilibrium by showing that the LQ game admits an -potential structure, with determined by the degree of pairwise interaction asymmetry. For pairwise-symmetric interactions, we construct an affine distributed equilibrium by minimizing the potential function and show that independent PG methods converge globally to this equilibrium, with complexity scaling linearly in the population size and logarithmically in the desired accuracy. For asymmetric interactions, we prove that independent projected PG algorithms converge linearly to an approximate equilibrium, with suboptimality proportional to the degree of asymmetry. Numerical experiments confirm the theoretical results across both symmetric and asymmetric interaction networks.
Paper Structure (30 sections, 19 theorems, 112 equations, 3 figures, 2 algorithms)

This paper contains 30 sections, 19 theorems, 112 equations, 3 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Independent learning with distributed policies.
  3. Our contributions.
  4. Challenges beyond existing works.
  5. Notation.
  6. Distributed equilibria for stochastic LQ games
  7. Mathematical setup
  8. $\alpha$-potential function
  9. Learning NEs under symmetric interactions
  10. Characterization of distributed NEs
  11. Independent policy gradient algorithm and its convergence
  12. Policy gradient algorithm
  13. Convergence analysis
  14. Learning approximate NEs under asymmetric interactions
  15. Characterization of distributed NEs
  16. ...and 15 more sections

Key Result

Proposition 2.3

The function $\Phi$ defined in eq:Phi_general_def is an $\alpha$-potential function of the game ${\mathbb{G}} = (I, (J^i)_{i \in I}, \tilde{\mathcal{V}})$, where and In particular, when $C_Q=0$, $\Phi$ is a potential function of the game ${\mathbb{G}} = (I, (J^i)_{i \in I}, {\mathcal{V}})$.

Figures (3)

  • Figure 1: Convergence of Algorithm \ref{['algo']} on symmetric uniform attachment networks.
  • Figure 2: Asymmetric Erdős--Rényi interaction networks with connection probability $p = 0.1$ (left), $p = 0.5$ (middle), and $p = 0.9$ (right).
  • Figure 3: Convergence of Algorithm \ref{['algo']} on asymmetric Erdős--Rényi networks.

Theorems & Definitions (44)

  • Remark 2.1
  • Definition 2.1
  • Definition 2.2
  • Proposition 2.3
  • Remark 2.2
  • Example 1
  • Example 2
  • Theorem 3.2
  • Remark 3.1
  • Lemma 3.3
  • ...and 34 more