Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploiting inter-agent coupling information for efficient reinforcement learning of cooperative LQR

Shahbaz P Qadri Syed, He Bai

TL;DR

This paper exploits inter-agent coupling information and proposes a systematic approach to exactly decompose the local Q-function of each agent, developing an approximate least square policy iteration algorithm and identifying two architectures to learn the local Q-function for each agent.

Abstract

Developing scalable and efficient reinforcement learning algorithms for cooperative multi-agent control has received significant attention over the past years. Existing literature has proposed inexact decompositions of local Q-functions based on empirical information structures between the agents. In this paper, we exploit inter-agent coupling information and propose a systematic approach to exactly decompose the local Q-function of each agent. We develop an approximate least square policy iteration algorithm based on the proposed decomposition and identify two architectures to learn the local Q-function for each agent. We establish that the worst-case sample complexity of the decomposition is equal to the centralized case and derive necessary and sufficient graphical conditions on the inter-agent couplings to achieve better sample efficiency. We demonstrate the improved sample efficiency and computational efficiency on numerical examples.

Exploiting inter-agent coupling information for efficient reinforcement learning of cooperative LQR

TL;DR

This paper exploits inter-agent coupling information and proposes a systematic approach to exactly decompose the local Q-function of each agent, developing an approximate least square policy iteration algorithm and identifying two architectures to learn the local Q-function for each agent.

Abstract

Developing scalable and efficient reinforcement learning algorithms for cooperative multi-agent control has received significant attention over the past years. Existing literature has proposed inexact decompositions of local Q-functions based on empirical information structures between the agents. In this paper, we exploit inter-agent coupling information and propose a systematic approach to exactly decompose the local Q-function of each agent. We develop an approximate least square policy iteration algorithm based on the proposed decomposition and identify two architectures to learn the local Q-function for each agent. We establish that the worst-case sample complexity of the decomposition is equal to the centralized case and derive necessary and sufficient graphical conditions on the inter-agent couplings to achieve better sample efficiency. We demonstrate the improved sample efficiency and computational efficiency on numerical examples.
Paper Structure (17 sections, 7 theorems, 55 equations, 1 figure, 3 tables, 1 algorithm)

This paper contains 17 sections, 7 theorems, 55 equations, 1 figure, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Formulation of the cooperative LQR problem
  3. Decomposition of the Q-function
  4. Multi-agent structured least square policy iteration
  5. Theoretical Analysis
  6. Simulations
  7. Conclusion
  8. Proof of Lemma \ref{['lm:rtc']}
  9. Proof of Theorem \ref{['thm:Qsetdecomp']}
  10. Proof of Theorem \ref{['thm:graddecomp']}
  11. Proof of Proposition \ref{['prop:linarch']}
  12. Necessary and sufficient graphical conditions for strictly better sample complexity
  13. Proof of Theorem \ref{['thm:undecsampcomp']}
  14. Analysis of the indirect case
  15. Remark on the sample complexity of the indirect case
  16. ...and 2 more sections

Key Result

Lemma 3.1

For any $i,j,k \in \mathcal{V}$, if $j\in \mathcal{I}^i_Q$, then for any $k\in \mathcal{R}^j_{SO}$, $k\in \mathcal{I}^i_Q$.

Figures (1)

  • Figure 1: Comparison of the total average cost for Example 1 (a) and Example 2 (b) using direct, indirect, undecomposed direct, and centralized Q-function architectures in Algorithm \ref{['alg:malspi']}.

Theorems & Definitions (14)

  • Lemma 3.1
  • Theorem 3.1: Value decomposition theorem
  • Theorem 3.2: Gradient decomposition theorem
  • Proposition 4.1
  • Theorem 5.1
  • proof
  • proof
  • proof
  • proof
  • Lemma E.1
  • ...and 4 more