Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Personalized Multi-Agent Average Reward TD-Learning via Joint Linear Approximation

Leo, Wang, Pengkun Yang, Lili Su

TL;DR

This work showed that this decomposition can filter out conflicting signals, effectively mitigating the negative impacts of ``misaligned''signals, and achieving linear speedup in cooperative single-timescale TD learning.

Abstract

We study personalized multi-agent average reward TD learning, in which a collection of agents interacts with different environments and jointly learns their respective value functions. We focus on the setting where there exists a shared linear representation, and the agents' optimal weights collectively lie in an unknown linear subspace. Inspired by the recent success of personalized federated learning (PFL), we study the convergence of cooperative single-timescale TD learning in which agents iteratively estimate the common subspace and local heads. We showed that this decomposition can filter out conflicting signals, effectively mitigating the negative impacts of ``misaligned'' signals, and achieving linear speedup. The main technical challenges lie in the heterogeneity, the Markovian sampling, and their intricate interplay in shaping error evolutions. Specifically, not only are the error dynamics of multiple variables closely interconnected, but there is also no direct contraction for the principal angle distance between the optimal subspace and the estimated subspace. We hope our analytical techniques can be useful to inspire research on deeper exploration into leveraging common structures. Experiments are provided to show the benefits of learning via a shared structure to the more general control problem.

Personalized Multi-Agent Average Reward TD-Learning via Joint Linear Approximation

TL;DR

This work showed that this decomposition can filter out conflicting signals, effectively mitigating the negative impacts of ``misaligned''signals, and achieving linear speedup in cooperative single-timescale TD learning.

Abstract

We study personalized multi-agent average reward TD learning, in which a collection of agents interacts with different environments and jointly learns their respective value functions. We focus on the setting where there exists a shared linear representation, and the agents' optimal weights collectively lie in an unknown linear subspace. Inspired by the recent success of personalized federated learning (PFL), we study the convergence of cooperative single-timescale TD learning in which agents iteratively estimate the common subspace and local heads. We showed that this decomposition can filter out conflicting signals, effectively mitigating the negative impacts of ``misaligned'' signals, and achieving linear speedup. The main technical challenges lie in the heterogeneity, the Markovian sampling, and their intricate interplay in shaping error evolutions. Specifically, not only are the error dynamics of multiple variables closely interconnected, but there is also no direct contraction for the principal angle distance between the optimal subspace and the estimated subspace. We hope our analytical techniques can be useful to inspire research on deeper exploration into leveraging common structures. Experiments are provided to show the benefits of learning via a shared structure to the more general control problem.
Paper Structure (28 sections, 17 theorems, 283 equations, 4 figures, 2 tables, 1 algorithm)

This paper contains 28 sections, 17 theorems, 283 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Setup and Preliminaries
  4. Algorithm: PMAAR-TD
  5. Convergence Analysis
  6. Main Convergence Results
  7. Experiments
  8. Numerical Experiments on Prediction Problems
  9. Comparison with two-timescale setting
  10. Application to Control Problems
  11. Model Design
  12. Environment Configurations
  13. Result
  14. Conclusion and Discussions
  15. Additional Related Work
  16. ...and 13 more sections

Key Result

Lemma 5.1

Let ${\mathbf{P}}^* = {\mathbf{B}}^* ({\mathbf{B}}^*)^{\top}$, and ${\mathbf{P}}_t = {\mathbf{B}}_t {\mathbf{B}}_t^{\top}$ for each $t$. Choose $\tau = \lceil 2\log_\rho(\zeta)\rceil$, $U_{\delta} U_{\omega} \zeta\le \frac{1}{2}$, where $U_{\delta} = 2U_r + 2U_{\omega}$, $\beta = c\zeta$ for arbitra where ${\mathcal{C}}_{X,1}(\tau^2), {\mathcal{C}}_{X,2}(\tau^2), {\mathcal{C}}_{X,3}(\tau^2), {\mat

Figures (4)

  • Figure 1: Value function convergence comparison on Acrobot.
  • Figure 2: Value function convergence speed comparison between single-timescale and two-timescale settings on Acrobot.
  • Figure 3: Geometry illustration.
  • Figure 4: Policy Learning Performance Comparison on Acrobot

Theorems & Definitions (30)

  • Lemma 5.1: (informal) Upper bound of local head errors
  • Lemma 5.2: Lower bound of local head errors
  • Lemma 5.3: Principal angle distance analysis
  • Lemma 5.4: Reward analysis
  • Theorem 1: Convergence (informal)
  • Proposition 1
  • Lemma 3.1: Perturbation of QR
  • proof
  • Lemma 3.2
  • proof
  • ...and 20 more