Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

One-Shot Averaging for Distributed TD($λ$) Under Markov Sampling

Haoxing Tian, Ioannis Ch. Paschalidis, Alex Olshevsky

TL;DR

A distributed setup for reinforcement learning, where each agent has a copy of the same Markov Decision Process but transitions are sampled from the corresponding Markov chain independently by each agent, is considered, which achieves a linear speedup for TD(<inline-formula> <tex-math notation="LaTeX">$\lambda $ </tex-math></inline-formula>), a family of popular methods for policy evaluation.

Abstract

We consider a distributed setup for reinforcement learning, where each agent has a copy of the same Markov Decision Process but transitions are sampled from the corresponding Markov chain independently by each agent. We show that in this setting, we can achieve a linear speedup for TD($λ$), a family of popular methods for policy evaluation, in the sense that $N$ agents can evaluate a policy $N$ times faster provided the target accuracy is small enough. Notably, this speedup is achieved by ``one shot averaging,'' a procedure where the agents run TD($λ$) with Markov sampling independently and only average their results after the final step. This significantly reduces the amount of communication required to achieve a linear speedup relative to previous work.

One-Shot Averaging for Distributed TD($λ$) Under Markov Sampling

TL;DR

A distributed setup for reinforcement learning, where each agent has a copy of the same Markov Decision Process but transitions are sampled from the corresponding Markov chain independently by each agent, is considered, which achieves a linear speedup for TD(<inline-formula> <tex-math notation="LaTeX"> </tex-math></inline-formula>), a family of popular methods for policy evaluation.

Abstract

We consider a distributed setup for reinforcement learning, where each agent has a copy of the same Markov Decision Process but transitions are sampled from the corresponding Markov chain independently by each agent. We show that in this setting, we can achieve a linear speedup for TD(), a family of popular methods for policy evaluation, in the sense that agents can evaluate a policy times faster provided the target accuracy is small enough. Notably, this speedup is achieved by ``one shot averaging,'' a procedure where the agents run TD() with Markov sampling independently and only average their results after the final step. This significantly reduces the amount of communication required to achieve a linear speedup relative to previous work.
Paper Structure (22 sections, 11 theorems, 93 equations, 2 figures, 2 algorithms)

This paper contains 22 sections, 11 theorems, 93 equations, 2 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Markov Decision Process (MDP)
  4. Value Function Approximation
  5. Distributed Model and Algorithm
  6. Markov Sampling and Mixing
  7. Convergence times for centralized TD(0) and TD($\lambda$)
  8. Main Result
  9. Proof of Theorem \ref{['t: td with markov']}
  10. Proof of Lemma \ref{['l: smallest eigenvalue of i.i.d. matrix']}
  11. Proof of Lemma \ref{['l: singular value of i.i.d. matrix']}
  12. Proof of Lemma \ref{['l: bounded optimal point']}
  13. Proof of Lemma \ref{['l: decaying random variable 2']}
  14. Proof of Theorem \ref{['t: td lambda with markov']}
  15. Proof of Lemma \ref{['l: smallest eigenvalue of i.i.d. matrix, lambda']}
  16. ...and 7 more sections

Key Result

Lemma 2.1

In TD(0) with the Markov sampling, suppose Assumptions a: input assumption, a: uniform mixing hold and $t_{\rm th} = \max\{\tau_{\rm mix}, \frac{18}{(1-\gamma)^2 \omega^2}-1\}$. For a decaying stepsize sequence $\alpha_t = 2/(\omega (t+1) (1-\gamma))$,

Figures (2)

  • Figure 1: Numerical results for TD(0)
  • Figure 2: Numerical results for TD($\lambda$)

Theorems & Definitions (22)

  • Lemma 2.1
  • Lemma 2.2
  • Theorem 3.1
  • Theorem 3.2
  • Lemma 4.1
  • Lemma 4.2
  • Lemma 4.3
  • Lemma 4.4
  • proof : Proof of Theorem \ref{['t: td with markov']}
  • proof
  • ...and 12 more