Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Statistical inference for Linear Stochastic Approximation with Markovian Noise

Sergey Samsonov, Marina Sheshukova, Eric Moulines, Alexey Naumov

TL;DR

This work derives non-asymptotic Berry–Esseen bounds for Polyak–Ruppert averaged iterates in Linear Stochastic Approximation under Markov noise, achieving a Kolmogorov distance rate of $O(n^{-1/4})$ to the Gaussian limit. The authors employ a Poisson decomposition and martingale Berry–Esseen techniques to bound the projected error and establish a finite-sample distributional approximation for LSA. They also prove non-asymptotic validity of a multiplier subsample bootstrap for constructing confidence intervals, with a coverage-approximation rate of $O(n^{-1/10})$, and recover the $O(n^{-1/8})$ rate for the overlapping batch means estimator of the asymptotic variance. The methodology applies to TD learning and informs practical choices for step sizes and bootstrap tuning, enabling reliable uncertainty quantification in SA with Markov noise. Potential extensions include multivariate, nonlinear SA and refined constants.

Abstract

In this paper we derive non-asymptotic Berry-Esseen bounds for Polyak-Ruppert averaged iterates of the Linear Stochastic Approximation (LSA) algorithm driven by the Markovian noise. Our analysis yields $\mathcal{O}(n^{-1/4})$ convergence rates to the Gaussian limit in the Kolmogorov distance. We further establish the non-asymptotic validity of a multiplier block bootstrap procedure for constructing the confidence intervals, guaranteeing consistent inference under Markovian sampling. Our work provides the first non-asymptotic guarantees on the rate of convergence of bootstrap-based confidence intervals for stochastic approximation with Markov noise. Moreover, we recover the classical rate of order $\mathcal{O}(n^{-1/8})$ up to logarithmic factors for estimating the asymptotic variance of the iterates of the LSA algorithm.

Statistical inference for Linear Stochastic Approximation with Markovian Noise

TL;DR

This work derives non-asymptotic Berry–Esseen bounds for Polyak–Ruppert averaged iterates in Linear Stochastic Approximation under Markov noise, achieving a Kolmogorov distance rate of to the Gaussian limit. The authors employ a Poisson decomposition and martingale Berry–Esseen techniques to bound the projected error and establish a finite-sample distributional approximation for LSA. They also prove non-asymptotic validity of a multiplier subsample bootstrap for constructing confidence intervals, with a coverage-approximation rate of , and recover the rate for the overlapping batch means estimator of the asymptotic variance. The methodology applies to TD learning and informs practical choices for step sizes and bootstrap tuning, enabling reliable uncertainty quantification in SA with Markov noise. Potential extensions include multivariate, nonlinear SA and refined constants.

Abstract

In this paper we derive non-asymptotic Berry-Esseen bounds for Polyak-Ruppert averaged iterates of the Linear Stochastic Approximation (LSA) algorithm driven by the Markovian noise. Our analysis yields convergence rates to the Gaussian limit in the Kolmogorov distance. We further establish the non-asymptotic validity of a multiplier block bootstrap procedure for constructing the confidence intervals, guaranteeing consistent inference under Markovian sampling. Our work provides the first non-asymptotic guarantees on the rate of convergence of bootstrap-based confidence intervals for stochastic approximation with Markov noise. Moreover, we recover the classical rate of order up to logarithmic factors for estimating the asymptotic variance of the iterates of the LSA algorithm.

Paper Structure

This paper contains 31 sections, 54 theorems, 349 equations, 2 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Accuracy of Gaussian approximation for LSA with Markov noise
  4. Discussion.
  5. Multiplier subsample bootstrap for LSA
  6. Discussion.
  7. Discussion.
  8. Application to the TD learning
  9. Conclusion
  10. Constants
  11. Extended version of \ref{['assum:step-size']}
  12. Proof of \ref{['th:normal_approximation_markov_with_sigma_n']}
  13. Proof of \ref{['cor:normal_approximation_markov']}
  14. Last iterate bound
  15. Proof of \ref{['prop:last_iterate_bound']}
  16. ...and 16 more sections

Key Result

Proposition 1

Let $-\bar{\mathbf{A}}$ be a Hurwitz matrix. Then for any $P = P^{\top} \succ 0$, there exists a unique matrix $Q = Q^{\top} \succ 0$, satisfying the Lyapunov equation $\bar{\mathbf{A}}^\top Q + Q \bar{\mathbf{A}} = P$. Moreover, setting where $\kappa_{Q} = \lambda_{\max}(Q)/\lambda_{\min}(Q)$, it holds for any $\alpha \in [0, \alpha_{\infty}]$ that $\alpha a \leq 1/2$, and

Theorems & Definitions (99)

  • Proposition 1: Proposition 1 in samsonov2024gaussian
  • Theorem 1
  • proof
  • Corollary 1
  • Remark 1
  • Proposition 2
  • Proposition 3
  • Corollary 2
  • Theorem 2
  • proof
  • ...and 89 more