Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Solving Finite-Horizon MDPs via Low-Rank Tensors

Sergio Rozada, Jose Luis Orejuela, Antonio G. Marques

TL;DR

This work addresses learning optimal policies for finite-horizon MDPs by representing time-varying value functions as low-rank tensors using PARAFAC. It develops an optimization framework based on Bellman equations and introduces convergent block-coordinate and gradient-based methods for known dynamics, along with stochastic variants for unknown dynamics that connect to temporal-difference learning. The authors prove convergence under appropriate assumptions and demonstrate substantial reductions in parameter count and computational load in grid-world and resource-allocation experiments, including wireless communication and battery charging scenarios. The results show that low-rank tensor representations enable scalable, model-free VF estimation with competitive performance, highlighting a path toward more data- and compute-efficient finite-horizon RL.

Abstract

We study the problem of learning optimal policies in finite-horizon Markov Decision Processes (MDPs) using low-rank reinforcement learning (RL) methods. In finite-horizon MDPs, the policies, and therefore the value functions (VFs) are not stationary. This aggravates the challenges of high-dimensional MDPs, as they suffer from the curse of dimensionality and high sample complexity. To address these issues, we propose modeling the VFs of finite-horizon MDPs as low-rank tensors, enabling a scalable representation that renders the problem of learning optimal policies tractable. We introduce an optimization-based framework for solving the Bellman equations with low-rank constraints, along with block-coordinate descent (BCD) and block-coordinate gradient descent (BCGD) algorithms, both with theoretical convergence guarantees. For scenarios where the system dynamics are unknown, we adapt the proposed BCGD method to estimate the VFs using sampled trajectories. Numerical experiments further demonstrate that the proposed framework reduces computational demands in controlled synthetic scenarios and more realistic resource allocation problems.

Solving Finite-Horizon MDPs via Low-Rank Tensors

TL;DR

This work addresses learning optimal policies for finite-horizon MDPs by representing time-varying value functions as low-rank tensors using PARAFAC. It develops an optimization framework based on Bellman equations and introduces convergent block-coordinate and gradient-based methods for known dynamics, along with stochastic variants for unknown dynamics that connect to temporal-difference learning. The authors prove convergence under appropriate assumptions and demonstrate substantial reductions in parameter count and computational load in grid-world and resource-allocation experiments, including wireless communication and battery charging scenarios. The results show that low-rank tensor representations enable scalable, model-free VF estimation with competitive performance, highlighting a path toward more data- and compute-efficient finite-horizon RL.

Abstract

We study the problem of learning optimal policies in finite-horizon Markov Decision Processes (MDPs) using low-rank reinforcement learning (RL) methods. In finite-horizon MDPs, the policies, and therefore the value functions (VFs) are not stationary. This aggravates the challenges of high-dimensional MDPs, as they suffer from the curse of dimensionality and high sample complexity. To address these issues, we propose modeling the VFs of finite-horizon MDPs as low-rank tensors, enabling a scalable representation that renders the problem of learning optimal policies tractable. We introduce an optimization-based framework for solving the Bellman equations with low-rank constraints, along with block-coordinate descent (BCD) and block-coordinate gradient descent (BCGD) algorithms, both with theoretical convergence guarantees. For scenarios where the system dynamics are unknown, we adapt the proposed BCGD method to estimate the VFs using sampled trajectories. Numerical experiments further demonstrate that the proposed framework reduces computational demands in controlled synthetic scenarios and more realistic resource allocation problems.
Paper Structure (21 sections, 64 equations, 4 figures, 3 algorithms)

This paper contains 21 sections, 64 equations, 4 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notation and tensor decomposition
  4. Finite-horizon MDPs
  5. Policy iteration via low-rank tensors
  6. Problem formulation
  7. Block-coordinate methods
  8. Leveraging gradient descent
  9. Stochastic policy iteration
  10. Stochastic problem formulation
  11. Stochastic block coordinate methods
  12. Temporal-difference methods
  13. Numerical Analysis
  14. Policy evaluation with known MDP
  15. Stochastic policy evaluation
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Picture (a) shows the rewards of the grid-world environment. The remaining pictures show in time-step $3$ the estimated VFs and policy by (b) infinite-horizon policy iteration, (c) finite-horizon policy iteration, and (d) BCD-PI. In finite-horizon settings, the VFs and the policy are time-dependent.
  • Figure 2: $\mathrm{NFE}$ between the tensor of optimal VFs obtained via policy iteration low-rank PARAFAC decomposition in the grid-like setup.
  • Figure 3: The figure shows results for BCD-PE and BCD-PI in the first row, and BCGD-PE and BCGD-PI in the second. The columns display: (i) PE convergence in terms of (a) NFE and (b) $L({\mathcal{Q}})$; and (ii) PI convergence in terms of (c) NFE and (d) empirical return.
  • Figure 4: The figure shows the average return for S-BCGD-PI and BCTD-PI against different baselines for (a) the wireless communications setup, and for (b) the battery charging setup. S-BCGD-PI and BCTD-PI converge faster than the baselines and require significantly less number of parameters.