Computing the Bias of Constant-step Stochastic Approximation with Markovian Noise

TL;DR

A method based on infinitesimal generator comparisons to study the bias of the algorithm, which is the expected difference between $\theta_n$ -- the value at iteration $n$ -- and $\theta^*$ -- the unique equilibrium of the corresponding ODE, is developed.

Abstract

We study stochastic approximation algorithms with Markovian noise and constant step-size $α$. We develop a method based on infinitesimal generator comparisons to study the bias of the algorithm, which is the expected difference between $θ_n$ -- the value at iteration $n$ -- and $θ^*$ -- the unique equilibrium of the corresponding ODE. We show that, under some smoothness conditions, this bias is of order $O(α)$. Furthermore, we show that the time-averaged bias is equal to $αV + O(α^2)$, where $V$ is a constant characterized by a Lyapunov equation, showing that $\mathbb{E}[\barθ_n] \approx θ^*+Vα+ O(α^2)$, where $\barθ_n=(1/n)\sum_{k=1}^nθ_k$ is the Polyak-Ruppert average. We also show that $\barθ_n$ converges with high probability around $θ^*+αV$. We illustrate how to combine this with Richardson-Romberg extrapolation to derive an iterative scheme with a bias of order $O(α^2)$.

Figures (6)

• Figure 1: Comparison of $\theta_n$, $\bar{\theta}_n$ and $\bar{\theta}_{n/2:n}$ for various $\alpha$.
• Figure 2: Illustration of the error of $\bar{\theta}_n := \frac{1}{N} \sum_{k=1}^{n} \theta_{k}$ for various values of $\alpha=0.02\times2^{-k}$ with $k\in\{0\dots5\}$ and of the error of the extrapolation \ref{['eq:extrapolation']} for $\alpha=0.01$ and $\alpha=0.005$.
• Figure 3: Overview of the proof. The dashed rectangles indicate the lemmas that are proven in Appendix \ref{['sec:proof_lemma']}.
• Figure 4: Illustration of the behavior of $\theta_n$, $\varphi_{n}(\theta_0)$ and $\varphi_{n-k}(\theta_k)$.
• Figure 5: Behavior of $\theta_n$, $\bar{\theta}_n$ and $\bar{\theta}_{n/2:n}$ for various values of $\alpha$. All $y$-axis have the same scale.

Theorems & Definitions (19)

• theorem 1
• theorem 2
• theorem 3
• proposition 1
• proof
• proposition 2
• proof : Proof of Proposition \ref{['prop:V']}
• lemma 1
• proof
• lemma 2