Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tensor and Matrix Low-Rank Value-Function Approximation in Reinforcement Learning

Sergio Rozada, Santiago Paternain, Antonio G. Marques

TL;DR

This work addresses the challenge of value-function approximation in RL when state-action spaces are high-dimensional. It introduces a non-parametric, online, model-free framework that enforces low-rank structure in the SA-VF, first with matrices and then with tensors via PARAFAC decomposition. The main contributions are a stochastic online matrix low-rank TD learning algorithm, a parallel tensor low-rank TD learning method, convergence analyses for the synchronous matrix case and its tensor extension, and extensive numerical experiments showing strong performance with far fewer parameters than classical deep methods. The results indicate substantial scalability benefits and faster convergence in high-dimensional RL problems, enabling practical VF estimation in complex environments. The approach offers interpretable, parameter-efficient alternatives to deep networks for value-based RL in large-scale settings.

Abstract

Value-function (VF) approximation is a central problem in Reinforcement Learning (RL). Classical non-parametric VF estimation suffers from the curse of dimensionality. As a result, parsimonious parametric models have been adopted to approximate VFs in high-dimensional spaces, with most efforts being focused on linear and neural-network-based approaches. Differently, this paper puts forth a a parsimonious non-parametric approach, where we use stochastic low-rank algorithms to estimate the VF matrix in an online and model-free fashion. Furthermore, as VFs tend to be multi-dimensional, we propose replacing the classical VF matrix representation with a tensor (multi-way array) representation and, then, use the PARAFAC decomposition to design an online model-free tensor low-rank algorithm. Different versions of the algorithms are proposed, their complexity is analyzed, and their performance is assessed numerically using standardized RL environments.

Tensor and Matrix Low-Rank Value-Function Approximation in Reinforcement Learning

TL;DR

This work addresses the challenge of value-function approximation in RL when state-action spaces are high-dimensional. It introduces a non-parametric, online, model-free framework that enforces low-rank structure in the SA-VF, first with matrices and then with tensors via PARAFAC decomposition. The main contributions are a stochastic online matrix low-rank TD learning algorithm, a parallel tensor low-rank TD learning method, convergence analyses for the synchronous matrix case and its tensor extension, and extensive numerical experiments showing strong performance with far fewer parameters than classical deep methods. The results indicate substantial scalability benefits and faster convergence in high-dimensional RL problems, enabling practical VF estimation in complex environments. The approach offers interpretable, parameter-efficient alternatives to deep networks for value-based RL in large-scale settings.

Abstract

Value-function (VF) approximation is a central problem in Reinforcement Learning (RL). Classical non-parametric VF estimation suffers from the curse of dimensionality. As a result, parsimonious parametric models have been adopted to approximate VFs in high-dimensional spaces, with most efforts being focused on linear and neural-network-based approaches. Differently, this paper puts forth a a parsimonious non-parametric approach, where we use stochastic low-rank algorithms to estimate the VF matrix in an online and model-free fashion. Furthermore, as VFs tend to be multi-dimensional, we propose replacing the classical VF matrix representation with a tensor (multi-way array) representation and, then, use the PARAFAC decomposition to design an online model-free tensor low-rank algorithm. Different versions of the algorithms are proposed, their complexity is analyzed, and their performance is assessed numerically using standardized RL environments.
Paper Structure (22 sections, 3 theorems, 47 equations, 7 figures, 2 tables)

This paper contains 22 sections, 3 theorems, 47 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Reinforcement Learning
  4. Low-rank and matrix completion
  5. Matrix low-rank for SA-VF estimation
  6. Low-rank in Bellman's equation
  7. TD-learning
  8. Matrix low-rank TD learning
  9. Low-rank SA-VF approximation meets linear methods
  10. Tensor low-rank for SA-VF estimation
  11. PARAFAC decomposition
  12. Tensor low-rank model in Bellman's equations
  13. Tensor low-rank TD learning
  14. Modifying the tensor low-rank stochastic algorithms
  15. Trading off matrix and tensor low-rank representations
  16. ...and 7 more sections

Key Result

Proposition 1

Let $K>0$ be a constant denoting the rank of the truncated SVD operator $\mathrm{TSVD}_K(\cdot)$, and let $N$ and $M$ be, respectively, the number of rows and columns of the matrix $\mathrm{unvec}({\mathbf q}^\pi)$. Then, upon the definition of with $\sigma_{K+1}(t)$ being the $K+1$th singular value of matrix $\mathrm{unvec}({\mathbf r} + \gamma {\mathbf P} \boldsymbol{\Pi} {\hat{\mathbf q} }^\pi

Figures (7)

  • Figure 1: Singular values (SV) of ${\mathbf Q}^*$ obtained via policy iteration in four standard RL problems: (a) Frozenlake, (b) Racetrack, (c) JacksCarRental, and (d) Taxi-v3. The energy of ${\mathbf Q}^*$ tends to concentrate on the main SVs.
  • Figure 2: Singular values (SV) of ${\hat{\mathbf Q} }$ obtained via $Q$-learning in four standard RL problems: (a) Pendulum, (b) Cartpole, (c) Mountain car, and (d) Acrobot. The energy of ${\hat{\mathbf Q} }$ tends to concentrate on the main SVs.
  • Figure 3: $\mathrm{NFE}$ between the tensor form of ${\hat{\mathbf Q} }$ obtained via $Q$-learning and its low-rank PARAFAC decomposition in four standard RL problems: (a) Pendulum, (b) Cartpole, (c) Mountain car, and (d) Acrobot. We observe how the error decreases as the rank of the approximation increases.
  • Figure 4: Median number of steps per episode in 4 standard RL problems: (a) Pendulum, (b) Cartpole, (c) Mountain car, and (d) Goddard rocket. The MLR and TLR algorithms are faster than DQN and SV-RL, especially when the batch size in DQN is large. It is also faster than $Q$-learning when the $Q$-matrix is large.
  • Figure 5: Distribution of the mean cumulative rewards per episode of $100$ agents in four standard RL problems: (a) Pendulum, (b) Cartpole, (c) Mountain car, and (d) Goddard rocket. On the one hand, the distributions of $Q$-learning are always shifted toward worse results. On the other hand, the dependency of DQN on the size of the ER buffer is clear.
  • ...and 2 more figures

Theorems & Definitions (5)

  • Proposition 1
  • proof
  • Lemma 1
  • proof
  • Corollary 1