Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimal and instance-dependent guarantees for Markovian linear stochastic approximation

Wenlong Mou, Ashwin Pananjady, Martin J. Wainwright, Peter L. Bartlett

TL;DR

This work provides non-asymptotic, instance-dependent guarantees for Markovian linear stochastic approximation, showing that averaged SA with Polyak–Ruppert averaging achieves an optimal $O\left( \tfrac{t_{\text{mix}} d}{n} \right)$-type rate up to polylog factors, while detailing how the local structure of the problem governs higher-order terms. It derives a local minimax lower bound and demonstrates that the averaged estimator attains instance-optimal performance, highlighting the practical relevance for hyperparameter tuning in TD$(\lambda)$ and related linear Z-estimation problems. The results apply to TD$\{0,\lambda\}$, sieve estimators, and autoregressive models, providing a principled framework for adaptive model selection and parameter tuning in Markovian settings. The paper also introduces bootstrapping techniques and surrogate processes to achieve sharp dimension-dependent bounds, contributing new analytic tools for stochastic approximation with dependent data. Overall, the findings offer both theoretical tightness and actionable guidance for policy evaluation and time-series estimation in stochastic, Markov-driven environments.

Abstract

We study stochastic approximation procedures for approximately solving a $d$-dimensional linear fixed point equation based on observing a trajectory of length $n$ from an ergodic Markov chain. We first exhibit a non-asymptotic bound of the order $t_{\mathrm{mix}} \tfrac{d}{n}$ on the squared error of the last iterate of a standard scheme, where $t_{\mathrm{mix}}$ is a mixing time. We then prove a non-asymptotic instance-dependent bound on a suitably averaged sequence of iterates, with a leading term that matches the local asymptotic minimax limit, including sharp dependence on the parameters $(d, t_{\mathrm{mix}})$ in the higher order terms. We complement these upper bounds with a non-asymptotic minimax lower bound that establishes the instance-optimality of the averaged SA estimator. We derive corollaries of these results for policy evaluation with Markov noise -- covering the TD($λ$) family of algorithms for all $λ\in [0, 1)$ -- and linear autoregressive models. Our instance-dependent characterizations open the door to the design of fine-grained model selection procedures for hyperparameter tuning (e.g., choosing the value of $λ$ when running the TD($λ$) algorithm).

Optimal and instance-dependent guarantees for Markovian linear stochastic approximation

TL;DR

This work provides non-asymptotic, instance-dependent guarantees for Markovian linear stochastic approximation, showing that averaged SA with Polyak–Ruppert averaging achieves an optimal -type rate up to polylog factors, while detailing how the local structure of the problem governs higher-order terms. It derives a local minimax lower bound and demonstrates that the averaged estimator attains instance-optimal performance, highlighting the practical relevance for hyperparameter tuning in TD and related linear Z-estimation problems. The results apply to TD, sieve estimators, and autoregressive models, providing a principled framework for adaptive model selection and parameter tuning in Markovian settings. The paper also introduces bootstrapping techniques and surrogate processes to achieve sharp dimension-dependent bounds, contributing new analytic tools for stochastic approximation with dependent data. Overall, the findings offer both theoretical tightness and actionable guidance for policy evaluation and time-series estimation in stochastic, Markov-driven environments.

Abstract

We study stochastic approximation procedures for approximately solving a -dimensional linear fixed point equation based on observing a trajectory of length from an ergodic Markov chain. We first exhibit a non-asymptotic bound of the order on the squared error of the last iterate of a standard scheme, where is a mixing time. We then prove a non-asymptotic instance-dependent bound on a suitably averaged sequence of iterates, with a leading term that matches the local asymptotic minimax limit, including sharp dependence on the parameters in the higher order terms. We complement these upper bounds with a non-asymptotic minimax lower bound that establishes the instance-optimality of the averaged SA estimator. We derive corollaries of these results for policy evaluation with Markov noise -- covering the TD() family of algorithms for all -- and linear autoregressive models. Our instance-dependent characterizations open the door to the design of fine-grained model selection procedures for hyperparameter tuning (e.g., choosing the value of when running the TD() algorithm).
Paper Structure (101 sections, 30 theorems, 398 equations)

This paper contains 101 sections, 30 theorems, 398 equations.

Table of Contents

  1. Introduction
  2. Contributions and organization
  3. Notation:
  4. Problem set-up
  5. Assumptions
  6. Conditions on Markov chain
  7. Tail conditions on noise
  8. Metric space conditions
  9. Lipschitz condition
  10. Some illustrative examples
  11. Examples of Markov chains
  12. Examples of linear $Z$-estimators
  13. Main results
  14. Instance-dependent upper bounds
  15. Instance-dependent bounds on the averaged iterates
  16. ...and 86 more sections

Key Result

Theorem 1

Under Assumptions assume-markov-mixing--assume:stationary-tail, suppose that we set the stepsize $\eta$ and burn-in parameter ${n_0}$ as $\eta = (c (\sigma_L^2 d + \gamma_{\tiny{\operatorname{max}}}^2) (1 - \kappa) n^2 t_{\mathrm{mix}} )^{-1/3}$ and ${n_0} = \frac{1}{2} n$, where $c$ is a suitably c

Theorems & Definitions (33)

  • Example 1: Approximate policy evaluation
  • Example 2: Policy evaluation with TD$(\lambda)$
  • Example 3: Parameter estimation in autoregressive models
  • Theorem 1
  • Corollary 1
  • Proposition 1
  • Theorem 2
  • Corollary 2
  • Corollary 3
  • Corollary 4
  • ...and 23 more