$O(1/k)$ Finite-Time Bound for Non-Linear Two-Time-Scale Stochastic Approximation

TL;DR

This work analyzes non-linear two-time-scale stochastic approximation with contractive operators and derives a finite-time mean-square bound of $\mathcal{O}(1/k)$ when the stepsizes are $\alpha_k,\beta_k = \mathcal{O}(1/k)$. The key technical idea is to rewrite the slower-time-scale noise as an averaged sequence $U_{k}$ and study the transformed iterate $z_k=y_k-U_k$, enabling decoupled recursive bounds and an induction-based proof of boundedness. The paper also provides a weaker but robust bound of $\mathcal{O}(1/k^{a})$ for $a\in(0.5,1)$ under $\alpha_k=\mathcal{O}(1/k^{a})$, $\beta_k=\mathcal{O}(1/k)$ with largely parameter-independent constants. These results apply to gradient-descent-ascent and two-time-scale Lagrangian optimization, broadening finite-time guarantees to non-linear settings and suggesting extensions to Markovian noise and high-probability bounds.

Abstract

Two-time-scale stochastic approximation is an algorithm with coupled iterations which has found broad applications in reinforcement learning, optimization and game control. While several prior works have obtained a mean square error bound of $O(1/k)$ for linear two-time-scale iterations, the best known bound in the non-linear contractive setting has been $O(1/k^{2/3})$. In this work, we obtain an improved bound of $O(1/k)$ for non-linear two-time-scale stochastic approximation. Our result applies to algorithms such as gradient descent-ascent and two-time-scale Lagrangian optimization. The key step in our analysis involves rewriting the original iteration in terms of an averaged noise sequence which decays sufficiently fast. Additionally, we use an induction-based approach to show that the iterates are bounded in expectation.

Theorems & Definitions (12)

• Theorem 1
• Theorem 2
• Lemma 1
• Lemma 2
• Lemma 3
• Lemma 4
• Lemma 5
• Lemma 6
• Lemma 7
• Lemma 8