Statistical Inference for Temporal Difference Learning with Linear Function Approximation

TL;DR

This work advances statistical inference for Temporal Difference learning with linear function approximation by establishing nonasymptotic, high-probability bounds that scale with the asymptotic variance, under weaker conditions than prior results, and by proving a high-dimensional Berry-Esseen bound with rate $O(T^{-1/3})$. It also introduces an online plug-in estimator for the asymptotic covariance, enabling finite-sample confident regions for the linear value-function parameters, with provable coverage guarantees. The authors further compare their Berry-Esseen results to recent work and demonstrate the practical viability of the online covariance estimator in providing accurate Gaussian approximations for TD-based policy evaluation in high dimensions. Simulations corroborate the theory across various settings, highlighting the method’s efficiency and accuracy for constructing both individual and simultaneous confidence sets. Overall, the paper provides a principled, scalable framework for uncertainty quantification in TD learning with linear function approximation.

Abstract

We investigate the statistical properties of Temporal Difference (TD) learning with Polyak-Ruppert averaging, arguably one of the most widely used algorithms in reinforcement learning, for the task of estimating the parameters of the optimal linear approximation to the value function. Assuming independent samples, we make three theoretical contributions that improve upon the current state-of-the-art results: (i) we derive sharper high probability convergence guarantees that depend explicitly on the asymptotic variance and hold under weaker conditions than those adopted in the literature; (ii) we establish refined high-dimensional Berry-Esseen bounds over the class of convex sets, achieving faster rates than the best known results, and (iii) we propose and analyze a novel, computationally efficient online plug-in estimator of the asymptotic covariance matrix. These results enable the construction of confidence regions and simultaneous confidence intervals for the linear parameters of the value function approximation, with guaranteed finite-sample coverage. We demonstrate the applicability of our theoretical findings through numerical experiments.

Figures (9)

• Figure 1: Emergence of the first entry of TD estimation error on an MDP with a 3-dimensional feature map.
• Figure 2: (a) Empirical $95\%$ quantiles of $\|\overline{\bm{\Delta}}_T\|_2$, and (b) distance between the empirical and asymptotic distributions of $\overline{\bm{\Delta}}_T$.
• Figure 3: The decay of the $95\%$ quantile of $\|\overline{\bm{\Delta}}_T\|_2^2$ when (a) $d = 3$, for different choices of $\alpha$; and (b) $\alpha = \frac{2}{3}$, for different choices of $d$.
• Figure 4: The decay of $d(\sqrt{T}\overline{\bm{\Delta}}_T,\mathcal{N}(\bm{0},\bm{\Lambda}^\star))$, when (a) $d=3$, for different choices of $\alpha$; and (b) $\alpha = \frac{2}{3}$, for different choices of $d$.
• Figure 5: The decay of the difference between estimated and asymptotic covariance matrices, measured in squared Frobenius norm, when (a) $d = 3$, for different choices of $\alpha$; and (b) $\alpha = \frac{2}{3}$, for different choices of $d$.

Theorems & Definitions (31)

• Theorem 3.1
• Theorem 3.2
• Theorem 3.3
• Theorem 3.4
• Lemma A.1
• Lemma A.2
• Lemma A.3
• Lemma A.4
• Lemma A.5
• Lemma A.6