Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

n-Step Temporal Difference Learning with Optimal n

Lakshmi Mandal, Shalabh Bhatnagar

TL;DR

This work tackles the problem of selecting the optimal number of steps $n$ in $n$-step TD to minimize the long-run RMSE, denoting this objective by $J(n)$. It introduces SDPSA, a one-simulation, deterministic perturbation SPSA-like algorithm operating on a two-timescale stochastic approximation: a fast TD($n$) estimator for fixed $n$, and a slow SDPSA-driven update of the discrete $n \\in D=\{1,...,L\}$. Theoretical guarantees are provided by formulating a differential inclusion (DI) with a Marchaud set-valued map $H(n)$ to handle discontinuities in $\dot{J}(n)$; the slower update converges almost surely to the limit points of this DI, identifying the optimal $n$ that minimizes the long-run variance of the TD estimator. Empirically, SDPSA converges from arbitrary initial $n$ on Random Walk and Grid World tasks and outperforms the state-of-the-art OCBA in RMSE and computation time. The approach offers a principled, model-free means to optimize discrete RL parameters and demonstrates the practicality of DI-based analysis for stochastic approximation in reinforcement learning.

Abstract

We consider the problem of finding the optimal value of n in the n-step temporal difference (TD) learning algorithm. Our objective function for the optimization problem is the average root mean squared error (RMSE). We find the optimal n by resorting to a model-free optimization technique involving a one-simulation simultaneous perturbation stochastic approximation (SPSA) based procedure. Whereas SPSA is a zeroth-order continuous optimization procedure, we adapt it to the discrete optimization setting by using a random projection operator. We prove the asymptotic convergence of the recursion by showing that the sequence of n-updates obtained using zeroth-order stochastic gradient search converges almost surely to an internally chain transitive invariant set of an associated differential inclusion. This results in convergence of the discrete parameter sequence to the optimal n in n-step TD. Through experiments, we show that the optimal value of n is achieved with our SDPSA algorithm for arbitrary initial values. We further show using numerical evaluations that SDPSA outperforms the state-of-the-art discrete parameter stochastic optimization algorithm Optimal Computing Budget Allocation (OCBA) on benchmark RL tasks.

n-Step Temporal Difference Learning with Optimal n

TL;DR

This work tackles the problem of selecting the optimal number of steps in -step TD to minimize the long-run RMSE, denoting this objective by . It introduces SDPSA, a one-simulation, deterministic perturbation SPSA-like algorithm operating on a two-timescale stochastic approximation: a fast TD() estimator for fixed , and a slow SDPSA-driven update of the discrete . Theoretical guarantees are provided by formulating a differential inclusion (DI) with a Marchaud set-valued map to handle discontinuities in ; the slower update converges almost surely to the limit points of this DI, identifying the optimal that minimizes the long-run variance of the TD estimator. Empirically, SDPSA converges from arbitrary initial on Random Walk and Grid World tasks and outperforms the state-of-the-art OCBA in RMSE and computation time. The approach offers a principled, model-free means to optimize discrete RL parameters and demonstrates the practicality of DI-based analysis for stochastic approximation in reinforcement learning.

Abstract

We consider the problem of finding the optimal value of n in the n-step temporal difference (TD) learning algorithm. Our objective function for the optimization problem is the average root mean squared error (RMSE). We find the optimal n by resorting to a model-free optimization technique involving a one-simulation simultaneous perturbation stochastic approximation (SPSA) based procedure. Whereas SPSA is a zeroth-order continuous optimization procedure, we adapt it to the discrete optimization setting by using a random projection operator. We prove the asymptotic convergence of the recursion by showing that the sequence of n-updates obtained using zeroth-order stochastic gradient search converges almost surely to an internally chain transitive invariant set of an associated differential inclusion. This results in convergence of the discrete parameter sequence to the optimal n in n-step TD. Through experiments, we show that the optimal value of n is achieved with our SDPSA algorithm for arbitrary initial values. We further show using numerical evaluations that SDPSA outperforms the state-of-the-art discrete parameter stochastic optimization algorithm Optimal Computing Budget Allocation (OCBA) on benchmark RL tasks.
Paper Structure (7 sections, 4 theorems, 45 equations, 17 figures, 2 tables, 2 algorithms)

This paper contains 7 sections, 4 theorems, 45 equations, 17 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. The Proposed Methodology
  3. $n$-step TD (TD($n$)) Algorithm
  4. The Proposed SDPSA Algorithm
  5. Asymptotic Convergence Analysis
  6. Experiments and Results
  7. Conclusions

Key Result

Lemma 1

$J(n)$ is a Lipschitz continuous function in $n\in \bar{D}$. Further, its derivative is piecewise Lipschitz continuous on intervals $[k,k+1)$, $1\leq k\leq L$ but discontinuous in general with points of discontinuity in the set $D$.

Figures (17)

  • Figure 1: Flowchart of the proposed algorithm.
  • Figure 2: Obtained RMSE for different values of $n$ and fixed $\alpha$ on RW.
  • Figure 3: Obtained RMSE for different values of $n$ and fixed $\alpha$ on GW.
  • Figure 4: (a) $n$-updates and (b) running RMSE values w.r.t. number of iterations, with initial $n=16$ and $\alpha=0.6$ on RW
  • Figure 5: (a) $n$-updates and (b) running RMSE values w.r.t. number of iterations, with initial $n=8$ and $\alpha=0.6$ on RW
  • ...and 12 more figures

Theorems & Definitions (10)

  • Lemma 1
  • proof
  • Proposition 1
  • proof
  • Definition 1
  • Lemma 2
  • proof
  • Theorem 3
  • proof
  • Remark 1