Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting a Design Choice in Gradient Temporal Difference Learning

Xiaochi Qian, Shangtong Zhang

TL;DR

The paper addresses instability from off-policy learning with function approximation by revisiting the ${A^\top}$TD design and introducing ${A^\top_t\text{TD}}$, a single-rate, memory-efficient off-policy policy evaluation method. It proves almost-sure convergence to the GTD TD fixed point and provides finite-sample guarantees with a projection variant that matches on-policy TD rates up to logarithmic factors. The method achieves ${O(K)\ln^2 t}$ memory and ${O(K)}$ per-step computation, offering practical tuning advantages over GTD's dual-learning-rate setup. Empirical results on standard benchmarks show competitive convergence and performance, supporting its potential as a robust alternative for off-policy evaluation in large-scale RL.

Abstract

Off-policy learning enables a reinforcement learning (RL) agent to reason counterfactually about policies that are not executed and is one of the most important ideas in RL. It, however, can lead to instability when combined with function approximation and bootstrapping, two arguably indispensable ingredients for large-scale reinforcement learning. This is the notorious deadly triad. The seminal work Sutton et al. (2008) pioneers Gradient Temporal Difference learning (GTD) as the first solution to the deadly triad, which has enjoyed massive success thereafter. During the derivation of GTD, some intermediate algorithm, called $A^\top$TD, was invented but soon deemed inferior. In this paper, we revisit this $A^\top$TD and prove that a variant of $A^\top$TD, called $A_t^\top$TD, is also an effective solution to the deadly triad. Furthermore, this $A_t^\top$TD only needs one set of parameters and one learning rate. By contrast, GTD has two sets of parameters and two learning rates, making it hard to tune in practice. We provide asymptotic analysis for $A^\top_t$TD and finite sample analysis for a variant of $A^\top_t$TD that additionally involves a projection operator. The convergence rate of this variant is on par with the canonical on-policy temporal difference learning.

Revisiting a Design Choice in Gradient Temporal Difference Learning

TL;DR

The paper addresses instability from off-policy learning with function approximation by revisiting the TD design and introducing , a single-rate, memory-efficient off-policy policy evaluation method. It proves almost-sure convergence to the GTD TD fixed point and provides finite-sample guarantees with a projection variant that matches on-policy TD rates up to logarithmic factors. The method achieves memory and per-step computation, offering practical tuning advantages over GTD's dual-learning-rate setup. Empirical results on standard benchmarks show competitive convergence and performance, supporting its potential as a robust alternative for off-policy evaluation in large-scale RL.

Abstract

Off-policy learning enables a reinforcement learning (RL) agent to reason counterfactually about policies that are not executed and is one of the most important ideas in RL. It, however, can lead to instability when combined with function approximation and bootstrapping, two arguably indispensable ingredients for large-scale reinforcement learning. This is the notorious deadly triad. The seminal work Sutton et al. (2008) pioneers Gradient Temporal Difference learning (GTD) as the first solution to the deadly triad, which has enjoyed massive success thereafter. During the derivation of GTD, some intermediate algorithm, called TD, was invented but soon deemed inferior. In this paper, we revisit this TD and prove that a variant of TD, called TD, is also an effective solution to the deadly triad. Furthermore, this TD only needs one set of parameters and one learning rate. By contrast, GTD has two sets of parameters and two learning rates, making it hard to tune in practice. We provide asymptotic analysis for TD and finite sample analysis for a variant of TD that additionally involves a projection operator. The convergence rate of this variant is on par with the canonical on-policy temporal difference learning.
Paper Structure (22 sections, 16 theorems, 154 equations, 2 figures)

This paper contains 22 sections, 16 theorems, 154 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Background
  3. ${\text{$A^\top_t$TD}}$: Revisiting the Design Choice of ${\text{$A^\top$TD}}$
  4. Asymptotic Convergence Analysis of ${\text{$A^\top_t$TD}}$
  5. Finite Sample Analysis of ${\text{$A^\top_t$TD}}$
  6. Experiments
  7. Related Work
  8. Conclusion
  9. Proof of Theorem \ref{['thm finite']}
  10. Auxiliary Lemmas
  11. Proof of Technical Lemmas
  12. Proof of Lemma \ref{['lem lr bounds']}
  13. Proof of Lemma \ref{['lem lr bounds 2']}
  14. Proof of Lemma \ref{['lem qm wt distance']}
  15. Proof of Lemma \ref{['lem gm1 bound']}
  16. ...and 7 more sections

Key Result

Theorem 1

Let Assumptions assu chain, assu feature, assu lr, & assu gap hold. Then the iterates $\qty{w_t}$ generated by eq direct gtd satisfies where $w_*$ is the TD fixed point defined in eq td fixed point.

Figures (2)

  • Figure 1: Comparison of \ref{['eq direct gtd']} with previous TD algorithms. All curves are averaged over 10 random seeds with shaded regions showing standard errors. The curves of Vtrace and HTD are invisible in Boyan's chain because they reduce to TD in the on-policy setting. The curves of Vtrace, HTD, and TD are almost invisible in Baird's counterexample because they diverge very quickly.
  • Figure 2: \ref{['eq direct gtd']} with different gap functions. All curves are averaged over 10 random seeds with shaded regions showing standard errors.

Theorems & Definitions (16)

  • Theorem 1
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • Lemma 6
  • Lemma 7
  • Lemma 8
  • Lemma 9
  • Lemma 10
  • ...and 6 more