Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unsynchronized Decentralized Q-Learning: Two Timescale Analysis By Persistence

Bora Yongacoglu, Gürdal Arslan, Serdar Yüksel

TL;DR

This work tackles non-stationarity in multi-agent reinforcement learning by removing the need for synchronized policy updates. It introduces an unsynchronized decentralized Q-learning algorithm that uses constant learning rates and inertia to tame moving-target effects in weakly acyclic stochastic games. The authors prove a high-probability convergence result: with appropriately small learning rates and sufficiently long exploration phases, the joint policy converges to stationary $\epsilon$-equilibria, even when agents update asynchronously. The approach extends regret-testing paradigms to unsynchronized settings and demonstrates practical viability through simulations, highlighting broader applicability to decentralized MARL with non-stationary environments.

Abstract

Non-stationarity is a fundamental challenge in multi-agent reinforcement learning (MARL), where agents update their behaviour as they learn. Many theoretical advances in MARL avoid the challenge of non-stationarity by coordinating the policy updates of agents in various ways, including synchronizing times at which agents are allowed to revise their policies. Synchronization enables analysis of many MARL algorithms via multi-timescale methods, but such synchronization is infeasible in many decentralized applications. In this paper, we study an unsynchronized variant of the decentralized Q-learning algorithm, a recent MARL algorithm for stochastic games. We provide sufficient conditions under which the unsynchronized algorithm drives play to equilibrium with high probability. Our solution utilizes constant learning rates in the Q-factor update, which we show to be critical for relaxing the synchronization assumptions of earlier work. Our analysis also applies to unsynchronized generalizations of a number of other algorithms from the regret testing tradition, whose performance is analyzed by multi-timescale methods that study Markov chains obtained via policy update dynamics. This work extends the applicability of the decentralized Q-learning algorithm and its relatives to settings in which parameters are selected in an independent manner, and tames non-stationarity without imposing the coordination assumptions of prior work.

Unsynchronized Decentralized Q-Learning: Two Timescale Analysis By Persistence

TL;DR

This work tackles non-stationarity in multi-agent reinforcement learning by removing the need for synchronized policy updates. It introduces an unsynchronized decentralized Q-learning algorithm that uses constant learning rates and inertia to tame moving-target effects in weakly acyclic stochastic games. The authors prove a high-probability convergence result: with appropriately small learning rates and sufficiently long exploration phases, the joint policy converges to stationary -equilibria, even when agents update asynchronously. The approach extends regret-testing paradigms to unsynchronized settings and demonstrates practical viability through simulations, highlighting broader applicability to decentralized MARL with non-stationary environments.

Abstract

Non-stationarity is a fundamental challenge in multi-agent reinforcement learning (MARL), where agents update their behaviour as they learn. Many theoretical advances in MARL avoid the challenge of non-stationarity by coordinating the policy updates of agents in various ways, including synchronizing times at which agents are allowed to revise their policies. Synchronization enables analysis of many MARL algorithms via multi-timescale methods, but such synchronization is infeasible in many decentralized applications. In this paper, we study an unsynchronized variant of the decentralized Q-learning algorithm, a recent MARL algorithm for stochastic games. We provide sufficient conditions under which the unsynchronized algorithm drives play to equilibrium with high probability. Our solution utilizes constant learning rates in the Q-factor update, which we show to be critical for relaxing the synchronization assumptions of earlier work. Our analysis also applies to unsynchronized generalizations of a number of other algorithms from the regret testing tradition, whose performance is analyzed by multi-timescale methods that study Markov chains obtained via policy update dynamics. This work extends the applicability of the decentralized Q-learning algorithm and its relatives to settings in which parameters are selected in an independent manner, and tames non-stationarity without imposing the coordination assumptions of prior work.
Paper Structure (25 sections, 6 theorems, 58 equations, 2 figures, 4 tables)

This paper contains 25 sections, 6 theorems, 58 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Organization
  3. Model
  4. Weakly Acyclic Stochastic Games
  5. Q-Functions in Stochastic Games
  6. Related Work
  7. Q-Learning and its Challenges
  8. Algorithms for Local Action Learners
  9. Regret Testers
  10. Unsynchronized Decentralized Q-Learning
  11. Assumptions
  12. Assumption on the transition kernel
  13. Assumptions on algorithmic parameters
  14. Assumptions on algorithmic randomness
  15. Theorem Statement
  16. ...and 10 more sections

Key Result

Theorem 1

\newlabeltheorem:main0 Let $\mathcal{G}$ be a weakly acyclic game and suppose each player $i \in \mathcal{N}$ uses Algorithm algo:main to play $\mathcal{G}$. Suppose Assumptions assumption:rho-delta, assumption:exploration-phases, and assumption:primitive-random-variables hold, and let $\epsilon > then ${\rm Pr} ( \bm{\phi}_t \in \bm{\Pi}^{0{\rm \text{-}eq}} _{SD} ) \geq 1 - \epsilon,$ for all

Figures (2)

  • Figure 1: Stage costs: Player 1 (respectively, 2) picks a row (column), and its stage cost is the 1st (2nd) entry of the chosen cell.
  • Figure 2: Frequency of $\bm{\phi}_t \in \bm{\Pi}^{0{\rm \text{-}eq}} _{SD}$, averaged over 500 trials.

Theorems & Definitions (21)

  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4
  • Definition 5
  • Definition 6
  • Definition 7
  • Definition 8
  • Definition 9
  • Theorem 1
  • ...and 11 more