Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decentralized Learning for Optimality in Stochastic Dynamic Teams and Games with Local Control and Global State Information

Bora Yongacoglu, Gürdal Arslan, Serdar Yüksel

TL;DR

The results presented here are the first, to the best of the authors' knowledge, to give formal guarantees of convergence to team optimality using independent learners in stochastic dynamic teams and common interest games.

Abstract

Stochastic dynamic teams and games are rich models for decentralized systems and challenging testing grounds for multi-agent learning. Previous work that guaranteed team optimality assumed stateless dynamics, or an explicit coordination mechanism, or joint-control sharing. In this paper, we present an algorithm with guarantees of convergence to team optimal policies in teams and common interest games. The algorithm is a two-timescale method that uses a variant of Q-learning on the finer timescale to perform policy evaluation while exploring the policy space on the coarser timescale. Agents following this algorithm are "independent learners": they use only local controls, local cost realizations, and global state information, without access to controls of other agents. The results presented here are the first, to our knowledge, to give formal guarantees of convergence to team optimality using independent learners in stochastic dynamic teams and common interest games.

Decentralized Learning for Optimality in Stochastic Dynamic Teams and Games with Local Control and Global State Information

TL;DR

The results presented here are the first, to the best of the authors' knowledge, to give formal guarantees of convergence to team optimality using independent learners in stochastic dynamic teams and common interest games.

Abstract

Stochastic dynamic teams and games are rich models for decentralized systems and challenging testing grounds for multi-agent learning. Previous work that guaranteed team optimality assumed stateless dynamics, or an explicit coordination mechanism, or joint-control sharing. In this paper, we present an algorithm with guarantees of convergence to team optimal policies in teams and common interest games. The algorithm is a two-timescale method that uses a variant of Q-learning on the finer timescale to perform policy evaluation while exploring the policy space on the coarser timescale. Agents following this algorithm are "independent learners": they use only local controls, local cost realizations, and global state information, without access to controls of other agents. The results presented here are the first, to our knowledge, to give formal guarantees of convergence to team optimality using independent learners in stochastic dynamic teams and common interest games.

Paper Structure

This paper contains 12 sections, 8 theorems, 75 equations, 3 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Notation
  4. Literature Review
  5. Background
  6. Stationary Markov Decision Problems and Q-learning
  7. Stochastic Games and Decentralized Q-learning
  8. Learning Team Optimality
  9. Beyond Team Optimality: Application to Weakly Acyclic Games
  10. Learning with Constant Aspirations
  11. A Simulation Study
  12. Concluding Remarks

Key Result

Lemma 1

In a common interest game, for all $i$, $\bm{\pi}^* \in \bm{\Pi}_{\rm opt}$, $\tilde{\bm{\pi}} \in \bm{\Pi} \setminus \bm{\Pi}_{\rm opt}$, we have

Figures (3)

  • Figure 1: Stage cost for a two-DM game where DM$^1$ (DM$^2$) chooses a row (a column) and its cost is the first (the second) entry in the chosen cell.
  • Figure 2: Stage cost for a two-DM game where DM$^1$ (DM$^2$) chooses a row (a column) and its cost is the first (the second) entry in the chosen cell.
  • Figure 3: Stage cost for a two-DM game where DM$^1$ (DM$^2$) chooses a row (a column) and its cost is the first (the second) entry in the chosen cell.

Theorems & Definitions (23)

  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4
  • Definition 5
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Theorem 1
  • ...and 13 more