Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

High-probability sample complexities for policy evaluation with linear function approximation

Gen Li, Weichen Wu, Yuejie Chi, Cong Ma, Alessandro Rinaldo, Yuting Wei

TL;DR

The paper studies policy evaluation with linear function approximation in discounted MDPs, delivering sharp, high-probability finite-sample guarantees for two core algorithms. In the on-policy setting, averaged TD with Polyak-Ruppert averaging achieves minimax-optimal dependence on the error tolerance epsilon and problem parameters, with a bound that scales as roughly O( max_s phi(s)^T Sigma^{-1} phi(s) (1+||theta^*||_{Sigma}^2) / ((1- abla)^2 epsilon^2) ) up to logarithmic factors. In the off-policy setting, the two-timescale TDC algorithm is shown to attain a high-probability bound with explicit dependence on rho_max, eigenvalues lambda_1, lambda_2, the feature covariance, and the target error, marking the first such bound for original TDC with minimax-optimal rate. A minimax lower bound demonstrates near-tightness of the TD bound in the on-policy case, while numerical experiments corroborate the theory, including stability of TDC on Baird’s counterexample and accelerated convergence of averaged TD. Overall, the results offer practical, parameter-aware guidance for sample-efficient policy evaluation under linear function approximation and highlight the fundamental limits via minimax analysis.

Abstract

This paper is concerned with the problem of policy evaluation with linear function approximation in discounted infinite horizon Markov decision processes. We investigate the sample complexities required to guarantee a predefined estimation error of the best linear coefficients for two widely-used policy evaluation algorithms: the temporal difference (TD) learning algorithm and the two-timescale linear TD with gradient correction (TDC) algorithm. In both the on-policy setting, where observations are generated from the target policy, and the off-policy setting, where samples are drawn from a behavior policy potentially different from the target policy, we establish the first sample complexity bound with high-probability convergence guarantee that attains the optimal dependence on the tolerance level. We also exhihit an explicit dependence on problem-related quantities, and show in the on-policy setting that our upper bound matches the minimax lower bound on crucial problem parameters, including the choice of the feature maps and the problem dimension.

High-probability sample complexities for policy evaluation with linear function approximation

TL;DR

The paper studies policy evaluation with linear function approximation in discounted MDPs, delivering sharp, high-probability finite-sample guarantees for two core algorithms. In the on-policy setting, averaged TD with Polyak-Ruppert averaging achieves minimax-optimal dependence on the error tolerance epsilon and problem parameters, with a bound that scales as roughly O( max_s phi(s)^T Sigma^{-1} phi(s) (1+||theta^*||_{Sigma}^2) / ((1- abla)^2 epsilon^2) ) up to logarithmic factors. In the off-policy setting, the two-timescale TDC algorithm is shown to attain a high-probability bound with explicit dependence on rho_max, eigenvalues lambda_1, lambda_2, the feature covariance, and the target error, marking the first such bound for original TDC with minimax-optimal rate. A minimax lower bound demonstrates near-tightness of the TD bound in the on-policy case, while numerical experiments corroborate the theory, including stability of TDC on Baird’s counterexample and accelerated convergence of averaged TD. Overall, the results offer practical, parameter-aware guidance for sample-efficient policy evaluation under linear function approximation and highlight the fundamental limits via minimax analysis.

Abstract

This paper is concerned with the problem of policy evaluation with linear function approximation in discounted infinite horizon Markov decision processes. We investigate the sample complexities required to guarantee a predefined estimation error of the best linear coefficients for two widely-used policy evaluation algorithms: the temporal difference (TD) learning algorithm and the two-timescale linear TD with gradient correction (TDC) algorithm. In both the on-policy setting, where observations are generated from the target policy, and the off-policy setting, where samples are drawn from a behavior policy potentially different from the target policy, we establish the first sample complexity bound with high-probability convergence guarantee that attains the optimal dependence on the tolerance level. We also exhihit an explicit dependence on problem-related quantities, and show in the on-policy setting that our upper bound matches the minimax lower bound on crucial problem parameters, including the choice of the feature maps and the problem dimension.
Paper Structure (87 sections, 12 theorems, 254 equations, 3 figures, 2 tables)

This paper contains 87 sections, 12 theorems, 254 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Two settings: on-policy vs. off-policy
  3. Our main contributions
  4. Other related works
  5. Finite-sample guarantees for policy evaluation
  6. Stochastic approximation
  7. Off-policy learning
  8. Notation
  9. Problem formulation
  10. Model and settings
  11. Markov decision process
  12. Linear approximation for the value function
  13. Policy evaluation with linear approximation
  14. On-policy evaluation with linear approximation
  15. Off-policy evaluation with linear approximation
  16. ...and 72 more sections

Key Result

Theorem 1

There exist universal, positive constants $C_{0},c_{0}>0$ and $c_{1}>0$ , such that for any given $0<\delta<1$, the averaged TD learning estimator $\overline{\bm{\theta}}_{T}$eq:TD-update-all after $T$ iterations satisfies the bound with probability at least $1-\delta$, provided that $\bm{\theta}_0 =\bm{0}$, $\eta_0 = \ldots = \eta_T = \eta <\frac{c_{0}(1-\gamma)}{\kappa\log(Td/\delta)}$ and

Figures (3)

  • Figure 1: (a) Comparisons of the estimation error of TD and averaged TD when $d=3$. (b) Comparisons of the estimation error for averaged TD with $d=3$ and $d=9$. Two curves in the middle represent their average errors, while the shaded areas represent the $95\%$ confidence bands.
  • Figure 2: Baird's counterexample. Taking action $a=1$ always leads to state $s = 7$, while taking $a = 0$ leads to one of the other six states with equal probability. The reward is set to be always zero.
  • Figure 3: Performances of off-policy averaged TD (red, $\eta = 0.02$) and TDC (blue, $\alpha = 0.02$, $\beta = 0.002$). Two curves in the middle represent their average errors, while the shaded areas correspond to $95\%$ confidence bands.

Theorems & Definitions (21)

  • Theorem 1
  • Corollary 1: Sample complexity of TD learning
  • Theorem 2: Minimax lower bound
  • Remark 1
  • Claim 1
  • Theorem 3
  • Remark 2
  • Corollary 2
  • Lemma 1
  • proof
  • ...and 11 more