Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decoupled Functional Central Limit Theorems for Two-Time-Scale Stochastic Approximation

Yuze Han, Xiang Li, Jiadong Liang, Zhihua Zhang

TL;DR

This work establishes decoupled functional central limit theorems for two-time-scale stochastic approximation, showing that rescaled trajectories of the fast and slow updates converge to stationary Ornstein–Uhlenbeck processes with dynamics governed by linear generators and Lyapunov covariances. The authors construct continuous-time trajectories by rescaling the inner and outer errors by the square roots of their step sizes and introduce an auxiliary sequence to mitigate cross-scale interference, enabling a clean diffusion limit. The results extend classical CLTs for SA to trajectory-level FCLTs in a two-time-scale setting and recover standard CLTs in special cases, with implications for precise stochastic behavior in bilevel optimization, TD learning, and related algorithms. The approach hinges on tightness, martingale problems, and careful handling of residual terms, providing a framework for online inference and future generalizations to Markovian or more intricate noise structures.

Abstract

In two-time-scale stochastic approximation (SA), two iterates are updated at different rates, governed by distinct step sizes, with each update influencing the other. Previous studies have demonstrated that the convergence rates of the error terms for these updates depend solely on their respective step sizes, a property known as decoupled convergence. However, a functional version of this decoupled convergence has not been explored. Our work fills this gap by establishing decoupled functional central limit theorems for two-time-scale SA, offering a more precise characterization of its asymptotic behavior. To achieve these results, we leverage the martingale problem approach and establish tightness as a crucial intermediate step. Furthermore, to address the interdependence between different time scales, we introduce an innovative auxiliary sequence to eliminate the primary influence of the fast-time-scale update on the slow-time-scale update.

Decoupled Functional Central Limit Theorems for Two-Time-Scale Stochastic Approximation

TL;DR

This work establishes decoupled functional central limit theorems for two-time-scale stochastic approximation, showing that rescaled trajectories of the fast and slow updates converge to stationary Ornstein–Uhlenbeck processes with dynamics governed by linear generators and Lyapunov covariances. The authors construct continuous-time trajectories by rescaling the inner and outer errors by the square roots of their step sizes and introduce an auxiliary sequence to mitigate cross-scale interference, enabling a clean diffusion limit. The results extend classical CLTs for SA to trajectory-level FCLTs in a two-time-scale setting and recover standard CLTs in special cases, with implications for precise stochastic behavior in bilevel optimization, TD learning, and related algorithms. The approach hinges on tightness, martingale problems, and careful handling of residual terms, providing a framework for online inference and future generalizations to Markovian or more intricate noise structures.

Abstract

In two-time-scale stochastic approximation (SA), two iterates are updated at different rates, governed by distinct step sizes, with each update influencing the other. Previous studies have demonstrated that the convergence rates of the error terms for these updates depend solely on their respective step sizes, a property known as decoupled convergence. However, a functional version of this decoupled convergence has not been explored. Our work fills this gap by establishing decoupled functional central limit theorems for two-time-scale SA, offering a more precise characterization of its asymptotic behavior. To achieve these results, we leverage the martingale problem approach and establish tightness as a crucial intermediate step. Furthermore, to address the interdependence between different time scales, we introduce an innovative auxiliary sequence to eliminate the primary influence of the fast-time-scale update on the slow-time-scale update.

Paper Structure

This paper contains 30 sections, 22 theorems, 116 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Notation and Organization
  5. Preliminaries
  6. Tightness and Weak Convergence
  7. The Martingale Problem Approach
  8. Assumptions
  9. Main Results
  10. Construction of the Continuous Trajectories
  11. Decoupled Functional Central Limit Theorems
  12. Examples
  13. Framework of Proof
  14. Derivation of One-Step Recursions and Introduction of the Auxiliary Sequence
  15. Establishment of Tightness
  16. ...and 15 more sections

Key Result

Proposition 2.1

Suppose that $\{ {\textnormal{U}}_{n}(\cdot) \}_{n=0}^\infty$ is a tight sequence of random functions in $C([0, \infty); {\mathbb R}^{d})$ and ${\textnormal{U}}(\cdot)$ is another random function in $C([0, \infty); {\mathbb R}^{d})$.

Figures (3)

  • Figure 1: Construction of ${\bar{\textnormal{X}}}_{n}(\cdot)$ and ${\bar{\textnormal{Y}}}_{n}(\cdot)$
  • Figure 2: Illustration for the framework of proof.
  • Figure 3: Illustration for the last step of proof.

Theorems & Definitions (46)

  • Definition 2.1: Tightness of random functions
  • Proposition 2.1: Properties of tightness
  • Definition 2.2: The martingale problem
  • Proposition 2.2: ethier2009markov
  • Remark 3.1
  • Example 3.1: Polynomially diminishing step sizes
  • Proposition 4.1: Non-asymptotic convergence rates, informal, han2024finite
  • Theorem 4.1: Decoupled functional central limit theorems
  • Remark 4.1: Explanation of Theorem \ref{['thm:fclt-twotime']}
  • Remark 4.2: Comparison with previous work
  • ...and 36 more