Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerated Distributional Temporal Difference Learning with Linear Function Approximation

Kaicheng Jin, Yang Peng, Jiansheng Yang, Zhihua Zhang

TL;DR

The paper tackles finite-sample analysis for distributional TD learning with linear function approximation under a linear-categorical parameterization. It develops a baseline Linear-CTD with an instance-dependent convergence theory and proves a minimax lower bound that motivates variance-reduction. The authors introduce VrFLCTD, a variance-reduced fast algorithm that achieves minimax-optimal, $K$-independent sample complexity in both generative and Markovian settings, thereby showing that learning the full return distribution can be as statistically efficient as learning its expectation. Empirical results on small MDPs and a 2D grid world corroborate the theory, illustrating robust performance as the support size grows and the horizon increases. These results advance the understanding of the statistical efficiency of distributional reinforcement learning with function approximation and offer practical algorithms for scalable uncertainty-aware policy evaluation.

Abstract

In this paper, we study the finite-sample statistical rates of distributional temporal difference (TD) learning with linear function approximation. The purpose of distributional TD learning is to estimate the return distribution of a discounted Markov decision process for a given policy. Previous works on statistical analysis of distributional TD learning focus mainly on the tabular case. We first consider the linear function approximation setting and conduct a fine-grained analysis of the linear-categorical Bellman equation. Building on this analysis, we further incorporate variance reduction techniques in our new algorithms to establish tight sample complexity bounds independent of the support size $K$ when $K$ is large. Our theoretical results imply that, when employing distributional TD learning with linear function approximation, learning the full distribution of the return function from streaming data is no more difficult than learning its expectation. This work provide new insights into the statistical efficiency of distributional reinforcement learning algorithms.

Accelerated Distributional Temporal Difference Learning with Linear Function Approximation

TL;DR

The paper tackles finite-sample analysis for distributional TD learning with linear function approximation under a linear-categorical parameterization. It develops a baseline Linear-CTD with an instance-dependent convergence theory and proves a minimax lower bound that motivates variance-reduction. The authors introduce VrFLCTD, a variance-reduced fast algorithm that achieves minimax-optimal, -independent sample complexity in both generative and Markovian settings, thereby showing that learning the full return distribution can be as statistically efficient as learning its expectation. Empirical results on small MDPs and a 2D grid world corroborate the theory, illustrating robust performance as the support size grows and the horizon increases. These results advance the understanding of the statistical efficiency of distributional reinforcement learning with function approximation and offer practical algorithms for scalable uncertainty-aware policy evaluation.

Abstract

In this paper, we study the finite-sample statistical rates of distributional temporal difference (TD) learning with linear function approximation. The purpose of distributional TD learning is to estimate the return distribution of a discounted Markov decision process for a given policy. Previous works on statistical analysis of distributional TD learning focus mainly on the tabular case. We first consider the linear function approximation setting and conduct a fine-grained analysis of the linear-categorical Bellman equation. Building on this analysis, we further incorporate variance reduction techniques in our new algorithms to establish tight sample complexity bounds independent of the support size when is large. Our theoretical results imply that, when employing distributional TD learning with linear function approximation, learning the full distribution of the return function from streaming data is no more difficult than learning its expectation. This work provide new insights into the statistical efficiency of distributional reinforcement learning algorithms.

Paper Structure

This paper contains 47 sections, 27 theorems, 186 equations, 2 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Organization
  4. Backgrounds
  5. Notation
  6. Preliminaries
  7. Distributional Policy Evaluation
  8. Observation model
  9. Linear-Categorical Parameterization
  10. Related Works
  11. Distributional Temporal Difference learning
  12. Stochastic Optimization with Variance-reduction
  13. Comparisons of Theoretical Results with our Previous Work
  14. Comparisons of Theoretical Results with the Conference Version
  15. Linear-Categorical Temporal Difference Learning
  16. ...and 32 more sections

Key Result

Proposition 1

Let where the expectation is taken on the tuple $(s,a,s^{\prime},r)$ drawn from the generative model. Then it holds that

Figures (2)

  • Figure 1: Convergence results in four-state MDP with different algorithms and observation settings. For this 4-state MDP with $K = 20$ and $\gamma = 0.99$, VrFLCTD shows better convergence performance than the baseline algorithm Linear-CTD in the long run.
  • Figure 2: Convergence results for 2D Grid World in the Markovian Setting for different $K$ and $\gamma$. While achieving similar convergence rate at early stage, VrFLCTD exhibits a notably more consistent and stable convergence rate in the long run. $\gamma$ closer to 1 leads to a slower convergence rate.

Theorems & Definitions (39)

  • Proposition 1
  • Proposition 2
  • Proposition 3
  • Corollary 1
  • Theorem 1
  • Remark 1
  • Theorem 2
  • Corollary 2
  • Lemma 1
  • Theorem 3
  • ...and 29 more