Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Concentration of Contractive Stochastic Approximation: Additive and Multiplicative Noise

Zaiwei Chen, Siva Theja Maguluri, Martin Zubeldia

TL;DR

This work studies stochastic approximation under a contractive operator with respect to a general norm and derives non-asymptotic, high-probability bounds for SA iterates under both multiplicative and additive sub-Gaussian noise. The authors develop a novel bootstrapping framework that bounds the moment-generating function of a modified generalized Moreau envelope and constructs an exponential supermartingale to apply Ville's maximal inequality, yielding maximal concentration bounds with Weibull tails for multiplicative noise and sub-Gaussian tails for additive noise. A key impossibility result shows that sub-exponential tails cannot generally be achieved under multiplicative noise, and the analysis extends to linear SA and RL algorithms such as TD-learning and Q-learning. The results provide precise convergence rates and tail behaviors, enabling finite-sample guarantees and informing practical performance guarantees for RL and large-scale SA methods.

Abstract

In this paper, we establish maximal concentration bounds for the iterates generated by a stochastic approximation (SA) algorithm under a contractive operator with respect to some arbitrary norm (for example, the $\ell_\infty$-norm). We consider two settings where the iterates are potentially unbounded: SA with bounded multiplicative noise and SA with sub-Gaussian additive noise. Our maximal concentration inequalities state that the convergence error has a sub-Gaussian tail in the additive noise setting and a Weibull tail (which is faster than polynomial decay but could be slower than exponential decay) in the multiplicative noise setting. In addition, we provide an impossibility result showing that it is generally impossible to have sub-exponential tails under multiplicative noise. To establish the maximal concentration bounds, we develop a novel bootstrapping argument that involves bounding the moment-generating function of a modified version of the generalized Moreau envelope of the convergence error and constructing an exponential supermartingale to enable using Ville's maximal inequality. We demonstrate the applicability of our theoretical results in the context of linear SA and reinforcement learning.

Concentration of Contractive Stochastic Approximation: Additive and Multiplicative Noise

TL;DR

This work studies stochastic approximation under a contractive operator with respect to a general norm and derives non-asymptotic, high-probability bounds for SA iterates under both multiplicative and additive sub-Gaussian noise. The authors develop a novel bootstrapping framework that bounds the moment-generating function of a modified generalized Moreau envelope and constructs an exponential supermartingale to apply Ville's maximal inequality, yielding maximal concentration bounds with Weibull tails for multiplicative noise and sub-Gaussian tails for additive noise. A key impossibility result shows that sub-exponential tails cannot generally be achieved under multiplicative noise, and the analysis extends to linear SA and RL algorithms such as TD-learning and Q-learning. The results provide precise convergence rates and tail behaviors, enabling finite-sample guarantees and informing practical performance guarantees for RL and large-scale SA methods.

Abstract

In this paper, we establish maximal concentration bounds for the iterates generated by a stochastic approximation (SA) algorithm under a contractive operator with respect to some arbitrary norm (for example, the -norm). We consider two settings where the iterates are potentially unbounded: SA with bounded multiplicative noise and SA with sub-Gaussian additive noise. Our maximal concentration inequalities state that the convergence error has a sub-Gaussian tail in the additive noise setting and a Weibull tail (which is faster than polynomial decay but could be slower than exponential decay) in the multiplicative noise setting. In addition, we provide an impossibility result showing that it is generally impossible to have sub-exponential tails under multiplicative noise. To establish the maximal concentration bounds, we develop a novel bootstrapping argument that involves bounding the moment-generating function of a modified version of the generalized Moreau envelope of the convergence error and constructing an exponential supermartingale to enable using Ville's maximal inequality. We demonstrate the applicability of our theoretical results in the context of linear SA and reinforcement learning.
Paper Structure (43 sections, 23 theorems, 150 equations, 3 figures, 1 table)

This paper contains 43 sections, 23 theorems, 150 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Challenges & Our Techniques
  3. Related Literature
  4. Main Results
  5. Stochastic Approximation with Multiplicative Noise
  6. Tightening the Bounds for Small Values of $k$
  7. An Impossibility Result on the Tail Decay Rate
  8. Stochastic Approximation with Sub-Gaussian Additive Noise
  9. Proof of Theorem \ref{['thm:multi']}
  10. Time-Varying Worst-Case Bounds
  11. An Iterative Framework to Improve the Bound
  12. Bounding the Log-MGFs
  13. An Exponential Supermartingale
  14. Completing the Bootstrapping Argument
  15. Proof of Theorem \ref{['thm:impossibility']}
  16. ...and 28 more sections

Key Result

Proposition 1.1

Given a tolerance level $\delta\in (0,1)$, suppose that there exists a non-decreasing sequence $\{T_k(\delta)\}$ such that $\mathbb{P}(\|x_k-x^*\|_c^2\leq T_k(\delta),\forall\;k\geq 0)\geq 1-\delta$. Then, for any $\delta'\in (0,1-\delta)$, there must exist a sequence $\{T_k(\delta,\delta')\}$ with

Figures (3)

  • Figure 1: For $D>0$, all the iterates lie in the blue shaded area with probability at least $1-\delta$.
  • Figure 2: Percentiles of the squared error for a 1-dimensional SA with $D>0$.
  • Figure 3: Best tail exponent in Eq. (\ref{['eq:example_bound1']}) (black) vs. upper bound on best possible tail exponent given by Theorem \ref{['thm:impossibility']} (1) (dashed blue)

Theorems & Definitions (33)

  • Proposition 1.1
  • Theorem 2.1
  • Corollary 2.2
  • Example 2.1
  • Example 2.2
  • Theorem 2.3
  • Theorem 2.4
  • Proposition 3.1
  • Remark
  • proof : Proof of Proposition \ref{['prop:worst_case_bound']}
  • ...and 23 more