Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi Timescale Stochastic Approximation: Stability and Convergence

Rohan Deb, Swetha Ganesh, Shalabh Bhatnagar

TL;DR

This work establishes a comprehensive set of sufficient conditions ensuring stability and almost-sure convergence for general $N$-timescale stochastic approximation recursions using the ODE method. It then applies the theory to reinforcement learning algorithms that incorporate momentum, off-policy learning, and primal–dual constraints, showing that all components—including auxiliary momentum states and dual variables—converge to their respective limiting objects without ad hoc stability bounds. The results yield explicit convergence targets, such as $\theta^* = -\bar{A}^{-1}\bar{b}$ in GTD-type settings and saddle points in constrained/dual formulations, under standard assumptions on Lipschitzness, noise, and step-sizes. The framework thereby unifies and guarantees the stability of diverse multi-timescale RL algorithms, enabling momentum acceleration, off-policy stability, and primal–dual optimization to be analyzed with provable convergence guarantees.

Abstract

This paper presents the first sufficient conditions that guarantee the stability and almost sure convergence of multi-timescale stochastic approximation (SA) iterates. It extends the existing results on one-timescale and two-timescale SA iterates to general $N$-timescale stochastic recursions, for any $N \geq 1$, using the ordinary differential equation (ODE) method. As an application, we study SA algorithms augmented with heavy-ball momentum in the context of Gradient Temporal Difference (GTD) learning. The added momentum introduces an auxiliary state evolving on an intermediate timescale, yielding a three-timescale recursion. We show that with appropriate momentum parameters, the scheme fits within our framework and converges almost surely to the same fixed point as baseline GTD. The stability and convergence of all iterates including the momentum state follow from our main results without ad hoc bounds. We then study off-policy actor-critic algorithms with a baseline learner, actor, and critic updated on separate timescales. In contrast to prior work, we eliminate projection steps from the actor update and instead use our framework to guarantee stability and almost sure convergence of all components. Finally, we extend the analysis to constrained policy optimization in the average reward setting, where the actor, critic, and dual variables evolve on three distinct timescales, and we verify that the resulting dynamics satisfy the conditions of our general theorem. These examples show how diverse reinforcement learning algorithms covering momentum acceleration, off-policy learning, and primal-dual methods-fit naturally into the proposed multi-timescale framework.

Multi Timescale Stochastic Approximation: Stability and Convergence

TL;DR

This work establishes a comprehensive set of sufficient conditions ensuring stability and almost-sure convergence for general -timescale stochastic approximation recursions using the ODE method. It then applies the theory to reinforcement learning algorithms that incorporate momentum, off-policy learning, and primal–dual constraints, showing that all components—including auxiliary momentum states and dual variables—converge to their respective limiting objects without ad hoc stability bounds. The results yield explicit convergence targets, such as in GTD-type settings and saddle points in constrained/dual formulations, under standard assumptions on Lipschitzness, noise, and step-sizes. The framework thereby unifies and guarantees the stability of diverse multi-timescale RL algorithms, enabling momentum acceleration, off-policy stability, and primal–dual optimization to be analyzed with provable convergence guarantees.

Abstract

This paper presents the first sufficient conditions that guarantee the stability and almost sure convergence of multi-timescale stochastic approximation (SA) iterates. It extends the existing results on one-timescale and two-timescale SA iterates to general -timescale stochastic recursions, for any , using the ordinary differential equation (ODE) method. As an application, we study SA algorithms augmented with heavy-ball momentum in the context of Gradient Temporal Difference (GTD) learning. The added momentum introduces an auxiliary state evolving on an intermediate timescale, yielding a three-timescale recursion. We show that with appropriate momentum parameters, the scheme fits within our framework and converges almost surely to the same fixed point as baseline GTD. The stability and convergence of all iterates including the momentum state follow from our main results without ad hoc bounds. We then study off-policy actor-critic algorithms with a baseline learner, actor, and critic updated on separate timescales. In contrast to prior work, we eliminate projection steps from the actor update and instead use our framework to guarantee stability and almost sure convergence of all components. Finally, we extend the analysis to constrained policy optimization in the average reward setting, where the actor, critic, and dual variables evolve on three distinct timescales, and we verify that the resulting dynamics satisfy the conditions of our general theorem. These examples show how diverse reinforcement learning algorithms covering momentum acceleration, off-policy learning, and primal-dual methods-fit naturally into the proposed multi-timescale framework.
Paper Structure (17 sections, 15 theorems, 113 equations, 2 figures, 1 table)

This paper contains 17 sections, 15 theorems, 113 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Notations and Assumptions
  3. The 3-Timescale Case
  4. The N-Timescale Case
  5. Main Results
  6. Applications of the General Theory
  7. Proof Sketch of Theorem \ref{['main_thm']}
  8. Proof of the Main Results
  9. Showing convergence by assuming Stability (Theorem \ref{['Thm_with_stability']})
  10. Showing Stability of the recursions (Theorem \ref{['thm_without_stability']})
  11. Application to Reinforcement Learning Algorithms
  12. Gradient Temporal Difference with Momentum
  13. Three Timescale Gradient TD Algorithms with Momentum
  14. Experiments
  15. Off-Policy Actor Critic
  16. ...and 2 more sections

Key Result

Theorem 1

We define for $1\leq j \leq N-2,$$\lambda^{(j:N-1)}(x) \stackrel{\triangle}{=} \lambda^{(j)}(\lambda^{(j+1)}(\ldots\lambda^{(N-2)}(\lambda^{(N-1)}(x),x),\ldots,x),x)$. Under the assumptions $\textbf{(A:1)-(A:3)}$, (B.N.i)$_{1\leq i \leq N}$ and (C.N.i)$_{1\leq i \leq N}$,

Figures (2)

  • Figure 1: 5-State Random Walk from FastGradient.
  • Figure 3: RMSPBE across episodes (averaged over 100 independent runs) for 5-State Random Walk. The features used are the Dependent features used in FastGradient.

Theorems & Definitions (17)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Remark 1
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • ...and 7 more