Logit-Q Dynamics for Efficient Learning in Stochastic Teams

TL;DR

This paper tackles efficient learning in stochastic teams with unknown dynamics by introducing logit-Q dynamics that couple log-linear learning with Q-learning in a stage-game decomposition of stochastic games. The authors develop four variants (AL-Q, EL-Q, AIL-Q, EIL-Q) that differ in action revision and joint-play estimation, and establish convergence to near-efficient equilibria with computable error bounds, even under non-stationarity of continuation payoffs. A novel epoch-based analysis and a coupling lemma handle nonstationarity and connect the evolving Q-functions to stationary-like behavior within epochs, yielding tracking guarantees for the estimated plays and convergence of Q-functions to the team optimum. The results extend to single-controller potential games and provide rationality guarantees against pure stationary strategies, with numerical illustrations showing the practical convergence properties and the advantages of exploration-free variants. Overall, the framework offers a principled, model-free approach to achieve efficient coordination in complex multi-agent environments where dynamics are unknown or non-stationary.

Abstract

We present a new family of logit-Q dynamics for efficient learning in stochastic games by combining the log-linear learning (also known as logit dynamics) for the repeated play of normal-form games with Q-learning for unknown Markov decision processes within the auxiliary stage-game framework. In this framework, we view stochastic games as agents repeatedly playing some stage game associated with the current state of the underlying game while the agents' Q-functions determine the payoffs of these stage games. We show that the logit-Q dynamics presented reach (near) efficient equilibrium in stochastic teams with unknown dynamics and quantify the approximation error. We also show the rationality of the logit-Q dynamics against agents following pure stationary strategies and the convergence of the dynamics in stochastic games where the stage-payoffs induce potential games, yet only a single agent controls the state transitions beyond stochastic teams. The key idea is to approximate the dynamics with a fictional scenario where the Q-function estimates are stationary over epochs whose lengths grow at a sufficiently slow rate. We then couple the dynamics in the main and fictional scenarios to show that these two scenarios become more and more similar across epochs due to the vanishing step size and growing epoch lengths.

Figures (2)

• Figure 1: Consider two states $s$ and $\overline{s}$ with action profiles $\{a,b\}$ and $\{\overline{a},\overline{b}\}$, respectively. This is a figurative illustration for the global trajectory of the pairs $(s_t,a_t)_{t\geq 0}$ evolving over the global timescale $t=0,1,\ldots$, and the local trajectory of the pairs $(w_{k},z_{k})_{k\geq 0}$ specific to $s$ and evolving over the local timescale $k=1,2,\ldots$ whenever $s$ gets visited. Let $t_k$ be the time of the $k$th visit to $s$. In the local trajectory, $w_k$ and $z_k$ denote, resp., the action profile played at the $k$th visit to $s$ and the trajectory of the state-action pairs realized in between the $k$th and $(k+1)$th visit to $s$.
• Figure 2: From top to bottom, the solid curves represent the AL-Q, EL-Q, AIL-Q, and EIL-Q dynamics, respectively. The shaded areas show the standard deviation across independent runs. In \ref{['fig:QQstar', 'fig:UUstar']}, the gap for Q-functions and game values converge to the (near) optimal ones for all dynamics, corroborating \ref{['thm:main']}. In \ref{['fig:QQ']}, the impact of different initializations on the dynamics' long-run behavior diminishes in time, showing robustness to arbitrary initializations.

Theorems & Definitions (23)

• Definition 1
• Theorem 1
• Corollary 1
• proof
• Corollary 2
• proof
• Proposition 1
• proof
• Lemma 1
• Remark 1