Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stochastic Approximation with Unbounded Markovian Noise: A General-Purpose Theorem

Shaan Ul Haque, Siva Theja Maguluri

TL;DR

This work develops a general-purpose theorem for nonlinear stochastic approximation (SA) under unbounded Markovian noise by leveraging a Lyapunov drift condition and the Poisson equation to decompose noise into a martingale component plus tractable residual terms. This framework yields finite-time, mean-square convergence bounds for SA with unbounded state spaces and Markovian dynamics, without requiring restrictive mixing, by embedding the problem in a projected Lyapunov analysis. The paper then demonstrates the theorem's power through four settings: average-reward TD with linear function approximation, generalized linear models with Markovian data using LMS, Q-learning in finite-state RL with improved bounds, and stochastic cyclic block coordinate descent for distributed optimization, deriving optimal $\mathcal{O}(1/\varepsilon^2)$ sample complexities in several cases. Collectively, these results enable robust, finite-time guarantees for RL and optimization algorithms operating under unbounded Markov noise, broadening applicability to infinite-state MDPs and high-dimensional, real-world systems.

Abstract

Motivated by engineering applications such as resource allocation in networks and inventory systems, we consider average-reward Reinforcement Learning with unbounded state space and reward function. Recent works studied this problem in the actor-critic framework and established finite sample bounds assuming access to a critic with certain error guarantees. We complement their work by studying Temporal Difference (TD) learning with linear function approximation and establishing finite-time bounds with the optimal $\mathcal{O}\left(1/ε^2\right)$ sample complexity. These results are obtained using the following general-purpose theorem for non-linear Stochastic Approximation (SA). Suppose that one constructs a Lyapunov function for a non-linear SA with certain drift condition. Then, our theorem establishes finite-time bounds when this SA is driven by unbounded Markovian noise under suitable conditions. It serves as a black box tool to generalize sample guarantees on SA from i.i.d. or martingale difference case to potentially unbounded Markovian noise. The generality and the mild assumption of the setup enables broad applicability of our theorem. We illustrate its power by studying two more systems: (i) We improve upon the finite-time bounds of $Q$-learning by tightening the error bounds and also allowing for a larger class of behavior policies. (ii) We establish the first ever finite-time bounds for distributed stochastic optimization of high-dimensional smooth strongly convex function using cyclic block coordinate descent.

Stochastic Approximation with Unbounded Markovian Noise: A General-Purpose Theorem

TL;DR

This work develops a general-purpose theorem for nonlinear stochastic approximation (SA) under unbounded Markovian noise by leveraging a Lyapunov drift condition and the Poisson equation to decompose noise into a martingale component plus tractable residual terms. This framework yields finite-time, mean-square convergence bounds for SA with unbounded state spaces and Markovian dynamics, without requiring restrictive mixing, by embedding the problem in a projected Lyapunov analysis. The paper then demonstrates the theorem's power through four settings: average-reward TD with linear function approximation, generalized linear models with Markovian data using LMS, Q-learning in finite-state RL with improved bounds, and stochastic cyclic block coordinate descent for distributed optimization, deriving optimal sample complexities in several cases. Collectively, these results enable robust, finite-time guarantees for RL and optimization algorithms operating under unbounded Markov noise, broadening applicability to infinite-state MDPs and high-dimensional, real-world systems.

Abstract

Motivated by engineering applications such as resource allocation in networks and inventory systems, we consider average-reward Reinforcement Learning with unbounded state space and reward function. Recent works studied this problem in the actor-critic framework and established finite sample bounds assuming access to a critic with certain error guarantees. We complement their work by studying Temporal Difference (TD) learning with linear function approximation and establishing finite-time bounds with the optimal sample complexity. These results are obtained using the following general-purpose theorem for non-linear Stochastic Approximation (SA). Suppose that one constructs a Lyapunov function for a non-linear SA with certain drift condition. Then, our theorem establishes finite-time bounds when this SA is driven by unbounded Markovian noise under suitable conditions. It serves as a black box tool to generalize sample guarantees on SA from i.i.d. or martingale difference case to potentially unbounded Markovian noise. The generality and the mild assumption of the setup enables broad applicability of our theorem. We illustrate its power by studying two more systems: (i) We improve upon the finite-time bounds of -learning by tightening the error bounds and also allowing for a larger class of behavior policies. (ii) We establish the first ever finite-time bounds for distributed stochastic optimization of high-dimensional smooth strongly convex function using cyclic block coordinate descent.

Paper Structure

This paper contains 49 sections, 38 theorems, 163 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Setting and Main Result
  4. Unbounded State Space
  5. Bounded State Space
  6. SA with almost sure convergence but diverging mean square error
  7. Policy Evaluation for Infinite State Space MDPs
  8. Average-Reward TD
  9. Properties of TD algorithm
  10. Finite Sample Bounds for TD
  11. Approximation Error due to LFA
  12. Special Case of Reversible Markov Chains
  13. Generalized Linear Model with Markovian data
  14. Properties of LMS algorithm
  15. Finite Sample Bounds for LMS algorithm
  16. ...and 34 more sections

Key Result

Theorem 3.1

Suppose that we run the Markov chain with initial state $y_0$. When the state space $\mathcal{Y}$ is unbounded and the set $\mathcal{X}$ is an $\ell_2$-ball, then under the Assumptions assump:linear_F-assump:step-size, $\{x_k\}_{k\geq 0}$ in the iterations eq:main_eq satisfy the following:

Figures (2)

  • Figure 1: $\mathcal{S}_1=\{\{Q_l\}_{l\geq 0}; Q_0=0, D_l=1~\forall 0\leq l\leq k_1\}$. For illustrative purposes, we represent $Q_l$ as a continuous piecewise linear function by linearly interpolating queue lengths between the time instants.
  • Figure 2: $\mathcal{S}_1=\{\{Q_l\}_{l\geq 0}; Q_0=0, D_l=1~\forall 0\leq l\leq k_1\}$, $\mathcal{S}_2=\{\{Q_l\}_{l\geq 0}; Q_l\geq q_0~\forall 0\leq l<\infty \}$. For illustrative purposes, we represent $Q_l$ as a continuous piecewise linear function by linearly interpolating queue lengths between the time instants.

Theorems & Definitions (89)

  • Remark
  • Remark
  • Remark
  • Remark
  • Theorem 3.1
  • Remark
  • Remark
  • Theorem 3.2
  • Remark
  • Theorem 3.3
  • ...and 79 more