Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Variance Minimization Approach to Temporal-Difference Learning

Xingguo Chen, Yu Gong, Shangdong Yang, Wenhao Wang

TL;DR

This paper introduces a variance minimization (VM) approach for value-based RL instead of error minimization, and proposed two objectives, the Variance of Bellman Error and the Variance of Projected Bellman Error, and derived the VMTD, VMTDC, and VMETD algorithms.

Abstract

Fast-converging algorithms are a contemporary requirement in reinforcement learning. In the context of linear function approximation, the magnitude of the smallest eigenvalue of the key matrix is a major factor reflecting the convergence speed. Traditional value-based RL algorithms focus on minimizing errors. This paper introduces a variance minimization (VM) approach for value-based RL instead of error minimization. Based on this approach, we proposed two objectives, the Variance of Bellman Error (VBE) and the Variance of Projected Bellman Error (VPBE), and derived the VMTD, VMTDC, and VMETD algorithms. We provided proofs of their convergence and optimal policy invariance of the variance minimization. Experimental studies validate the effectiveness of the proposed algorithms.

A Variance Minimization Approach to Temporal-Difference Learning

TL;DR

This paper introduces a variance minimization (VM) approach for value-based RL instead of error minimization, and proposed two objectives, the Variance of Bellman Error and the Variance of Projected Bellman Error, and derived the VMTD, VMTDC, and VMETD algorithms.

Abstract

Fast-converging algorithms are a contemporary requirement in reinforcement learning. In the context of linear function approximation, the magnitude of the smallest eigenvalue of the key matrix is a major factor reflecting the convergence speed. Traditional value-based RL algorithms focus on minimizing errors. This paper introduces a variance minimization (VM) approach for value-based RL instead of error minimization. Based on this approach, we proposed two objectives, the Variance of Bellman Error (VBE) and the Variance of Projected Bellman Error (VPBE), and derived the VMTD, VMTDC, and VMETD algorithms. We provided proofs of their convergence and optimal policy invariance of the variance minimization. Experimental studies validate the effectiveness of the proposed algorithms.

Paper Structure

This paper contains 19 sections, 5 theorems, 99 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Markov Decision Process
  4. On-policy and Off-policy Learning
  5. Variance Minimization Algorithms
  6. Variance Minimization TD Learning: VMTD
  7. Variance Minimization TDC Learning: VMTDC
  8. Variance Minimization ETD Learning: VMETD
  9. Theoretical Analysis
  10. On the Fixed-point Solutions
  11. Experimental Studies
  12. Testing Tasks
  13. Experimental Results and Analysis
  14. Conclusion and Future Work
  15. Relevant proofs
  16. ...and 4 more sections

Key Result

Theorem 1

(The main factor affecting convergence rates chen2024convergence). Assume the same parameters setting for each algorithm, from the perspective of the expected convergence rate, the main factor that affects the convergence rate is the minimum eigenvalue of the matrix $\frac{1}{2}({\textbf{A}+\textbf{

Figures (1)

  • Figure 1: Learning curses of one evaluation environment and four control environments.

Theorems & Definitions (12)

  • Theorem 1
  • Theorem 2
  • proof
  • Theorem 3
  • proof
  • Theorem 4
  • proof
  • Theorem 5
  • proof
  • proof
  • ...and 2 more