Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gaussian Approximation for Two-Timescale Linear Stochastic Approximation

Bogdan Butyrin, Artemy Rubtsov, Alexey Naumov, Vladimir Ulyanov, Sergey Samsonov

TL;DR

This work establishes non-asymptotic Gaussian approximation rates for linear two-timescale stochastic approximation, addressing both martingale-difference and Markov noise. Using a decomposition framework that splits the statistic into a linear part plus a small nonlinear remainder, the authors prove high-order moment bounds and derive explicit convex-distance rates for both the Polyak–Ruppert averaged TTSA and the last iterate, with rates up to n^{-1/4} in the martingale case and n^{-1/6} under Markov noise. A key insight is the distinct influence of timescale separation on the last iterate versus averaging, and the analysis extends to TTSA implementations like GTD and TDC in reinforcement learning. These results enable non-asymptotic statistical inference for TTSA-based algorithms and pave the way for confidence-interval construction and tighter optimality bounds in practice.

Abstract

In this paper, we establish non-asymptotic bounds for accuracy of normal approximation for linear two-timescale stochastic approximation (TTSA) algorithms driven by martingale difference or Markov noise. Focusing on both the last iterate and Polyak-Ruppert averaging regimes, we derive bounds for normal approximation in terms of the convex distance between probability distributions. Our analysis reveals a non-trivial interaction between the fast and slow timescales: the normal approximation rate for the last iterate improves as the timescale separation increases, while it decreases in the Polyak-Ruppert averaged setting. We also provide the high-order moment bounds for the error of linear TTSA algorithm, which may be of independent interest.

Gaussian Approximation for Two-Timescale Linear Stochastic Approximation

TL;DR

This work establishes non-asymptotic Gaussian approximation rates for linear two-timescale stochastic approximation, addressing both martingale-difference and Markov noise. Using a decomposition framework that splits the statistic into a linear part plus a small nonlinear remainder, the authors prove high-order moment bounds and derive explicit convex-distance rates for both the Polyak–Ruppert averaged TTSA and the last iterate, with rates up to n^{-1/4} in the martingale case and n^{-1/6} under Markov noise. A key insight is the distinct influence of timescale separation on the last iterate versus averaging, and the analysis extends to TTSA implementations like GTD and TDC in reinforcement learning. These results enable non-asymptotic statistical inference for TTSA-based algorithms and pave the way for confidence-interval construction and tighter optimality bounds in practice.

Abstract

In this paper, we establish non-asymptotic bounds for accuracy of normal approximation for linear two-timescale stochastic approximation (TTSA) algorithms driven by martingale difference or Markov noise. Focusing on both the last iterate and Polyak-Ruppert averaging regimes, we derive bounds for normal approximation in terms of the convex distance between probability distributions. Our analysis reveals a non-trivial interaction between the fast and slow timescales: the normal approximation rate for the last iterate improves as the timescale separation increases, while it decreases in the Polyak-Ruppert averaged setting. We also provide the high-order moment bounds for the error of linear TTSA algorithm, which may be of independent interest.

Paper Structure

This paper contains 48 sections, 47 theorems, 450 equations.

Table of Contents

  1. Introduction
  2. Related works
  3. Gaussian Approximation for SA algorithms
  4. GAR for TTSA with Martingale noise
  5. Assumptions and definitions.
  6. Moment bounds for Martingale TTSA
  7. Discussion.
  8. GAR for Polyak-Ruppert averaged TTSA
  9. Proof sketch.
  10. Discussion.
  11. GAR for the last iterate.
  12. Discussion
  13. GAR for TTSA with Markov noise
  14. Moment bounds for TTSA with Markov noise
  15. Proof sketch.
  16. ...and 33 more sections

Key Result

Proposition 1

Let $\nu$ be a standard Gaussian measure in $\mathbb{R}^d$. Then for any random vectors $W, D$ taking values in $\mathbb{R}^d$, and any $p \geq 1$, where $c_d$ is the isoperimetric constant of class $\operatorname{Conv}(\mathbb{R}^{d})$.

Theorems & Definitions (87)

  • Proposition 1: Proposition 2 in sheshukova2025gaussian
  • Proposition 2: Observation 1 in kaledin2020finite
  • Proposition 3
  • Theorem 1
  • Theorem 2
  • Proposition 4
  • Theorem 3
  • Theorem 4
  • Lemma 1: Proposition A.1 in nourdin2022multivariate
  • Remark 1
  • ...and 77 more