Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prelimit Coupling and Steady-State Convergence of Constant-stepsize Nonsmooth Contractive SA

Yixuan Zhang, Dongyan Huo, Yudong Chen, Qiaomin Xie

TL;DR

This paper analyzes nonsmooth contractive stochastic approximation with constant stepsize, focusing on additive-noise SA and Q-learning (with multiplicative noise). It introduces a prelimit coupling technique to establish weak convergence to a unique stationary distribution in Wasserstein-2 and to characterize steady-state behavior as the stepsize vanishes. A central finding is that the asymptotic bias scales as $\sqrt{\alpha}$ in the nonsmooth setting, in contrast to smooth SA, and higher-order bias control is achieved via Richardson-Romberg extrapolation. The results are complemented by a diffusion-inspired analysis using the generalized Moreau envelope, explicit moment bounds, and concrete rate results, including an $\mathcal{O}(\alpha^{1/4})$ steady-state convergence rate for Q-learning; PR tail averaging and RR extrapolation are shown to effectively reduce bias and improve MSE in practice. These insights offer a principled path to bias mitigation in constant-stepsize nonsmooth SA and suggest applicability to broader nonsmooth stochastic-dynamical systems.

Abstract

Motivated by Q-learning, we study nonsmooth contractive stochastic approximation (SA) with constant stepsize. We focus on two important classes of dynamics: 1) nonsmooth contractive SA with additive noise, and 2) synchronous and asynchronous Q-learning, which features both additive and multiplicative noise. For both dynamics, we establish weak convergence of the iterates to a stationary limit distribution in Wasserstein distance. Furthermore, we propose a prelimit coupling technique for establishing steady-state convergence and characterize the limit of the stationary distribution as the stepsize goes to zero. Using this result, we derive that the asymptotic bias of nonsmooth SA is proportional to the square root of the stepsize, which stands in sharp contrast to smooth SA. This bias characterization allows for the use of Richardson-Romberg extrapolation for bias reduction in nonsmooth SA.

Prelimit Coupling and Steady-State Convergence of Constant-stepsize Nonsmooth Contractive SA

TL;DR

This paper analyzes nonsmooth contractive stochastic approximation with constant stepsize, focusing on additive-noise SA and Q-learning (with multiplicative noise). It introduces a prelimit coupling technique to establish weak convergence to a unique stationary distribution in Wasserstein-2 and to characterize steady-state behavior as the stepsize vanishes. A central finding is that the asymptotic bias scales as in the nonsmooth setting, in contrast to smooth SA, and higher-order bias control is achieved via Richardson-Romberg extrapolation. The results are complemented by a diffusion-inspired analysis using the generalized Moreau envelope, explicit moment bounds, and concrete rate results, including an steady-state convergence rate for Q-learning; PR tail averaging and RR extrapolation are shown to effectively reduce bias and improve MSE in practice. These insights offer a principled path to bias mitigation in constant-stepsize nonsmooth SA and suggest applicability to broader nonsmooth stochastic-dynamical systems.

Abstract

Motivated by Q-learning, we study nonsmooth contractive stochastic approximation (SA) with constant stepsize. We focus on two important classes of dynamics: 1) nonsmooth contractive SA with additive noise, and 2) synchronous and asynchronous Q-learning, which features both additive and multiplicative noise. For both dynamics, we establish weak convergence of the iterates to a stationary limit distribution in Wasserstein distance. Furthermore, we propose a prelimit coupling technique for establishing steady-state convergence and characterize the limit of the stationary distribution as the stepsize goes to zero. Using this result, we derive that the asymptotic bias of nonsmooth SA is proportional to the square root of the stepsize, which stands in sharp contrast to smooth SA. This bias characterization allows for the use of Richardson-Romberg extrapolation for bias reduction in nonsmooth SA.
Paper Structure (91 sections, 30 theorems, 368 equations, 2 figures)

This paper contains 91 sections, 30 theorems, 368 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Challenges of Applying Existing Techniques to Nonsmooth SA
  3. Prelimit Coupling Technique
  4. Step 1: Gaussian Noise and Rational Stepsize
  5. Step 2: General Stepsize
  6. Step 3: General Noise
  7. Notations
  8. Nonsmooth Stochastic Approximation with Additive Noise
  9. Model Setup
  10. Moments Bounds and Convergence to Stationary Distribution
  11. Steady-State Convergence and Bias Characterization
  12. Q-learning: Nonsmooth Stochastic Approximation with Multiplicative Noise
  13. Model Setup
  14. Moments Bounds and Convergence to Stationary Distribution
  15. Steady-State Convergence and Bias Characterization
  16. ...and 76 more sections

Key Result

Proposition 1

For each integer $n \geq 1$, under Assumption as: contraction0 and Assumption as: additive noise(n), there exists $\bar{\alpha}>0$ such that for any $\alpha \leq \bar{\alpha}$, there exists $t_{\alpha,n} > 0$ and where $c_n$ and $c_n^\prime$ are constants that are independent with $\alpha$ and $t$. Moreover, $t_{\alpha,1} = 0.$

Figures (2)

  • Figure 1: Steady-state convergence.
  • Figure 2: The errors of tail-averaged (TA) and RR extrapolated iterates with different stepsizes $\alpha$. In the legends, $\alpha=x$ RR means RR extrapolation with two stepsizes $x$ and $2x$.

Theorems & Definitions (40)

  • Proposition 1: Moment Bounds
  • Theorem 1: Distributional Convergence
  • Definition 1
  • Theorem 2: Steady-State Convergence
  • Theorem 3: Bias Characterization
  • Proposition 2: Moment Bounds
  • Theorem 4: Distributional Convergence
  • Theorem 5: Steady-State Convergence
  • Theorem 6: Bias Characterization
  • Proposition 3
  • ...and 30 more