Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Thompson Sampling Algorithm for Stochastic Games

Asaf Cohen, Ruolan He, Yuqiong Wang

TL;DR

The paper tackles learning in a continuous-time N-player stochastic differential game with ergodic quadratic costs and a common but unknown drift matrix. It develops a continuous-time Thompson Sampling algorithm with episodic posterior sampling, proves a Bayesian regret of $O(\sqrt{T \log T})$ independent of the number of players, and establishes that the TS-driven strategies form a Nash equilibrium by coupling TS dynamics with the full-information baseline. Theoretical results are complemented by numerical experiments demonstrating sublinear regret, convergence to equilibrium, and robustness to horizon length, dimension, and prior misspecification. This work advances equilibrium analysis under partial information in multi-agent continuous-time settings and offers practical learning-based methods for near-optimal strategic behavior in stochastic environments.

Abstract

We study a stochastic differential game with $N$ competitive players in a linear-quadratic framework with ergodic cost, where $d$-dimensional diffusion processes govern the state dynamics with an unknown common drift (matrix). Assuming a Gaussian prior on the drift, we use filtering techniques to update its posterior estimates. Based on these estimates, we propose a Thompson-sampling-based algorithm with dynamic episode lengths to approximate strategies. We show that the Bayesian regret for each player has an error bound of order $O(\sqrt{T\log(T)})$, where $T$ is the time-horizon, independent of the number of players. This implies that average regret per unit time goes to zero. Finally, we prove that the algorithm results in a Nash equilibrium.

Thompson Sampling Algorithm for Stochastic Games

TL;DR

The paper tackles learning in a continuous-time N-player stochastic differential game with ergodic quadratic costs and a common but unknown drift matrix. It develops a continuous-time Thompson Sampling algorithm with episodic posterior sampling, proves a Bayesian regret of independent of the number of players, and establishes that the TS-driven strategies form a Nash equilibrium by coupling TS dynamics with the full-information baseline. Theoretical results are complemented by numerical experiments demonstrating sublinear regret, convergence to equilibrium, and robustness to horizon length, dimension, and prior misspecification. This work advances equilibrium analysis under partial information in multi-agent continuous-time settings and offers practical learning-based methods for near-optimal strategic behavior in stochastic environments.

Abstract

We study a stochastic differential game with competitive players in a linear-quadratic framework with ergodic cost, where -dimensional diffusion processes govern the state dynamics with an unknown common drift (matrix). Assuming a Gaussian prior on the drift, we use filtering techniques to update its posterior estimates. Based on these estimates, we propose a Thompson-sampling-based algorithm with dynamic episode lengths to approximate strategies. We show that the Bayesian regret for each player has an error bound of order , where is the time-horizon, independent of the number of players. This implies that average regret per unit time goes to zero. Finally, we prove that the algorithm results in a Nash equilibrium.
Paper Structure (36 sections, 13 theorems, 209 equations, 11 figures, 1 algorithm)

This paper contains 36 sections, 13 theorems, 209 equations, 11 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. The ergodic $N$-player game with full information
  3. N-player ergodic game
  4. Learning for the $N$-player game
  5. Thompson Sampling algorithm
  6. Main results
  7. Examples
  8. Examples
  9. Symmetric system
  10. Nondefective system
  11. Numerical Experiments
  12. Baseline Setup
  13. The scaling behavior of the regret
  14. Comparison with CE and blind sampling
  15. The regret in higher dimensions
  16. ...and 21 more sections

Key Result

Proposition 2.1

Let Assumption assumption_merged be in force. Then, the HJB-KFP equation system HJB-KFP admits a unique solution $(v^i, m^i, \lambda^i)_{i \in [N]}$ with a quadratic function $v^i$, and it is of the form where $\Upsilon^i$ is the unique solution of the Riccati Equation eq_ARE which is a real symmetric positive definite matrix, $\Lambda^i = R^i(\varsigma^i \Upsilon^i + A)$, $\eta = (\eta^1, \ldots

Figures (11)

  • Figure 1: Cumulative and normalized regret for Thompson Sampling over time.
  • Figure 2: Cumulative and normalized regret over a long time horizon ($T=1000$).
  • Figure 3: Comparison of cumulative and normalized regret between TS and CE.
  • Figure 4: Comparison of cumulative and normalized regret between TS and Blind Sampling.
  • Figure 5: Cumulative and normalized regret over a long time horizon ($T=1000$).
  • ...and 6 more figures

Theorems & Definitions (20)

  • Definition 2.1
  • Definition 2.2
  • Remark 2.1
  • Proposition 2.1: Theorem 3.1 in bar-pri2014
  • Remark 3.1
  • Definition 3.1
  • Remark 3.2
  • Proposition 3.1: Posterior distribution
  • Theorem 3.1
  • Proposition 3.2
  • ...and 10 more