Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improved High-Probability Bounds for the Temporal Difference Learning Algorithm via Exponential Stability

Sergey Samsonov, Daniil Tiapkin, Alexey Naumov, Eric Moulines

TL;DR

The paper delivers sharp non-asymptotic guarantees for TD learning with linear function approximation by recasting TD as a linear stochastic approximation problem and applying Polyak-Ruppert tail averaging with a universal step size. A central innovation is an exponential stability framework for products of random matrices, enabling high-probability bounds and explicit sample complexity for TD(0) under both i.i.d. and Markov noise without projection steps. The authors show near-optimal variance and bias behavior, with instance-dependent rates that align with minimax limits in the i.i.d. setting, and extend the analysis to Markov noise via a data-drop TD_skip variant that preserves meaningful convergence guarantees tied to the mixing time. Overall, the work provides a principled, instance-general approach to TD analysis that improves practical applicability by avoiding problem-dependent step sizes and projection machinery while delivering rigorous performance guarantees. The results have implications for understanding TD reliability in real-world RL, where sampling often deviates from ideal i.i.d. assumptions and system conditioning plays a significant role.

Abstract

In this paper we consider the problem of obtaining sharp bounds for the performance of temporal difference (TD) methods with linear function approximation for policy evaluation in discounted Markov decision processes. We show that a simple algorithm with a universal and instance-independent step size together with Polyak-Ruppert tail averaging is sufficient to obtain near-optimal variance and bias terms. We also provide the respective sample complexity bounds. Our proof technique is based on refined error bounds for linear stochastic approximation together with the novel stability result for the product of random matrices that arise from the TD-type recurrence.

Improved High-Probability Bounds for the Temporal Difference Learning Algorithm via Exponential Stability

TL;DR

The paper delivers sharp non-asymptotic guarantees for TD learning with linear function approximation by recasting TD as a linear stochastic approximation problem and applying Polyak-Ruppert tail averaging with a universal step size. A central innovation is an exponential stability framework for products of random matrices, enabling high-probability bounds and explicit sample complexity for TD(0) under both i.i.d. and Markov noise without projection steps. The authors show near-optimal variance and bias behavior, with instance-dependent rates that align with minimax limits in the i.i.d. setting, and extend the analysis to Markov noise via a data-drop TD_skip variant that preserves meaningful convergence guarantees tied to the mixing time. Overall, the work provides a principled, instance-general approach to TD analysis that improves practical applicability by avoiding problem-dependent step sizes and projection machinery while delivering rigorous performance guarantees. The results have implications for understanding TD reliability in real-world RL, where sampling often deviates from ideal i.i.d. assumptions and system conditioning plays a significant role.

Abstract

In this paper we consider the problem of obtaining sharp bounds for the performance of temporal difference (TD) methods with linear function approximation for policy evaluation in discounted Markov decision processes. We show that a simple algorithm with a universal and instance-independent step size together with Polyak-Ruppert tail averaging is sufficient to obtain near-optimal variance and bias terms. We also provide the respective sample complexity bounds. Our proof technique is based on refined error bounds for linear stochastic approximation together with the novel stability result for the product of random matrices that arise from the TD-type recurrence.
Paper Structure (23 sections, 17 theorems, 166 equations, 1 table, 2 algorithms)

This paper contains 23 sections, 17 theorems, 166 equations, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related works.
  4. Notations.
  5. General LSA results
  6. Refined LSA results with i.i.d. noise
  7. TD learning under i.i.d. noise
  8. Comparison to the robust SA approach.
  9. Discussion and comparison.
  10. Discussion and comparison.
  11. On optimality of TD(0) for i.i.d. sampling scheme
  12. TD learning under Markov noise
  13. Discussion
  14. Conclusion
  15. Proofs of LSA error bounds presented in \ref{['sec:lsa']}
  16. ...and 8 more sections

Key Result

theorem 1

Assume assum:noise-level and assum:exp_stability($2$). Then for any $n \geq 2$, $\alpha \in (0;\alpha_{2,\infty}]$, and $\theta_0 \in \mathbb{R}^d$, it holds that

Theorems & Definitions (30)

  • theorem 1
  • theorem 2
  • theorem 3
  • theorem 4
  • theorem 5
  • theorem 6
  • theorem 7
  • proof
  • theorem 8
  • proof
  • ...and 20 more