Tight Finite Time Bounds of Two-Time-Scale Linear Stochastic Approximation with Markovian Noise

TL;DR

The paper provides tight finite-time covariance bounds for linear two-time-scale stochastic approximation with Markovian noise, showing that the leading mean-square error term for the slower variable scales as $\mathbb{E}[\hat{y}_k\hat{y}_k^T]=\beta_k\Sigma^y+o(\beta_k)$. The covariance matrices $\Sigma^x$, $\Sigma^{xy}$, and $\Sigma^y$ are characterized via coupled Lyapunov equations, yielding asymptotic normality results consistent with recent CLT analyses. A comprehensive step-size regime with $\alpha_k=\alpha/(k+K_0)^\xi$, $0.5<\xi<1$, and $\beta_k=\beta/(k+K_0)$ is shown to guarantee convergence, with the fast-time-scale choice of $\beta$ optimizing the asymptotic covariance. The results are specialized to Polyak-Ruppert averaging and RL off-policy algorithms (TD, TDC, GTD, GTD2), providing tight finite-time sample complexity bounds and practical guidance for step-size selection in these settings. Together, the work delivers a principled, general framework for finite-time analysis of two-time-scale SA under Markov noise with direct implications for stability and performance in RL methods.

Abstract

Stochastic approximation (SA) is an iterative algorithm for finding the fixed point of an operator using noisy samples and widely used in optimization and Reinforcement Learning (RL). The noise in RL exhibits a Markovian structure, and in some cases, such as gradient temporal difference (GTD) methods, SA is employed in a two-time-scale framework. This combination introduces significant theoretical challenges for analysis. We derive an upper bound on the error for the iterations of linear two-time-scale SA with Markovian noise. We demonstrate that the mean squared error decreases as $trace (Σ^y)/k + o(1/k)$ where $k$ is the number of iterates, and $Σ^y$ is an appropriately defined covariance matrix. A key feature of our bounds is that the leading term, $Σ^y$, exactly matches with the covariance in the Central Limit Theorem (CLT) for the two-time-scale SA, and we call them tight finite-time bounds. We illustrate their use in RL by establishing sample complexity for off-policy algorithms, TDC, GTD, and GTD2. A special case of linear two-time-scale SA that is extensively studied is linear SA with Polyak-Ruppert averaging. We present tight finite time bounds corresponding to the covariance matrix of the CLT. Such bounds can be used to study TD-learning with Polyak-Ruppert averaging.

Figures (5)

• Figure 1: Convergence behaviour of $\mathcal{E}_k$ for various choices of $\xi$ and $\beta$, where $\mathcal{E}_k=\frac{\|\hat{y}_k\hat{y}_k^\top\|}{\beta_k}$. The bold lines show the mean behavior across 5 sample paths, while the shaded region is the standard deviation from the mean. Both plots show a transition from stability to divergence of $\mathcal{E}_k$ when $\xi$ or $\beta$ do not satisfy the assumption \ref{['ass:step_size_main']}.
• Figure 2: The relationship among $\mathcal{A}, \mathcal{B}, \mathcal{C}, \mathcal{D}$ as 4 sets of the linear equations of the form \ref{['eq:lin_main']}.
• Figure 3: Divergence of two-time-scale linear SA when $\alpha_k=\alpha\beta_k/\beta$ and the ratio $\alpha/\beta$ is not carefully chosen.
• Figure 4: Road map of the proof of the paper
• Figure 5: The function $h(\beta)=\frac{\beta^2}{2\beta-1}$

Theorems & Definitions (69)

• Definition 4.1
• Proposition 4.1
• Theorem 4.1
• Proposition 4.2
• Corollary 4.1.1
• Remark
• Remark
• Proposition 4.3
• Proposition 4.4
• Theorem 4.2