Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Weighted mesh algorithms for general Markov decision processes: Convergence and tractability

Denis Belomestny, John Schoenmakers

TL;DR

The paper tackles high-dimensional finite-horizon MDPs with general (possibly noncompact) state spaces by introducing a weighted mesh/backward-simulation algorithm. It proves convergence in $L_1$ under compactness and extends to noncompact settings via domain reflection, providing explicit complexity bounds that are polynomial in the horizon and, in compact cases, tractable in the Novak–Wozniakowski sense; in noncompact cases the bounds are semi-tractable. Gaussian and LQG examples illustrate the method’s effectiveness and practical applicability to continuous-state problems. The results highlight how backward stochastic mesh contributions can mitigate the curse of dimensionality for a broad class of finite-horizon MDPs and offer guidance on algorithmic design and density-regularity conditions. Overall, the work presents a theoretically grounded, scalable framework for approximate dynamic programming in high-dimensional, general MDPs with potential for practical deployment in control and decision-making under uncertainty.

Abstract

We introduce a mesh-type approach for tackling discrete-time, finite-horizon Markov Decision Processes (MDPs) characterized by state and action spaces that are general, encompassing both finite and infinite (yet suitably regular) subsets of Euclidean space. In particular, for bounded state and action spaces, our algorithm achieves a computational complexity that is tractable in the sense of Novak and Wozniakowski, and is polynomial in the time horizon. For unbounded state space the algorithm is "semi-tractable" in the sense that the complexity is proportional to $ε^{-c}$ with some dimension independent $c\geq2$, for achieving an accuracy $ε$, and polynomial in the time horizon with degree linear in the underlying dimension. As such the proposed approach has some flavor of the randomization method by Rust which deals with infinite horizon MDPs and uniform sampling in compact state space. However, the present approach is essentially different due to the finite horizon and a simulation procedure due to general transition distributions, and more general in the sense that it encompasses unbounded state space. To demonstrate the effectiveness of our algorithm, we provide illustrations based on Linear-Quadratic Gaussian (LQG) control problems.

Weighted mesh algorithms for general Markov decision processes: Convergence and tractability

TL;DR

The paper tackles high-dimensional finite-horizon MDPs with general (possibly noncompact) state spaces by introducing a weighted mesh/backward-simulation algorithm. It proves convergence in under compactness and extends to noncompact settings via domain reflection, providing explicit complexity bounds that are polynomial in the horizon and, in compact cases, tractable in the Novak–Wozniakowski sense; in noncompact cases the bounds are semi-tractable. Gaussian and LQG examples illustrate the method’s effectiveness and practical applicability to continuous-state problems. The results highlight how backward stochastic mesh contributions can mitigate the curse of dimensionality for a broad class of finite-horizon MDPs and offer guidance on algorithmic design and density-regularity conditions. Overall, the work presents a theoretically grounded, scalable framework for approximate dynamic programming in high-dimensional, general MDPs with potential for practical deployment in control and decision-making under uncertainty.

Abstract

We introduce a mesh-type approach for tackling discrete-time, finite-horizon Markov Decision Processes (MDPs) characterized by state and action spaces that are general, encompassing both finite and infinite (yet suitably regular) subsets of Euclidean space. In particular, for bounded state and action spaces, our algorithm achieves a computational complexity that is tractable in the sense of Novak and Wozniakowski, and is polynomial in the time horizon. For unbounded state space the algorithm is "semi-tractable" in the sense that the complexity is proportional to with some dimension independent , for achieving an accuracy , and polynomial in the time horizon with degree linear in the underlying dimension. As such the proposed approach has some flavor of the randomization method by Rust which deals with infinite horizon MDPs and uniform sampling in compact state space. However, the present approach is essentially different due to the finite horizon and a simulation procedure due to general transition distributions, and more general in the sense that it encompasses unbounded state space. To demonstrate the effectiveness of our algorithm, we provide illustrations based on Linear-Quadratic Gaussian (LQG) control problems.
Paper Structure (16 sections, 6 theorems, 108 equations, 1 figure, 4 tables, 1 algorithm)

This paper contains 16 sections, 6 theorems, 108 equations, 1 figure, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Setup and basic properties of the Markov Decision Process
  3. Algorithm
  4. Convergence analysis
  5. Non-compact case
  6. Complexity
  7. Complexity for compact $\mathsf{S}$ and $\mathsf{A}$
  8. Complexity for noncompact $\mathsf{S}$ and compact $\mathsf{A}$. In particular, $\mathsf{A}$ can be infinite.
  9. Example: Gaussian transition densities
  10. Linear-quadratic Gaussian (LQG) control problems
  11. Proofs
  12. Proof of Proposition \ref{['prop:noncomp-compl']}
  13. Some auxiliary notions
  14. Estimation of mean uniformly in parameter
  15. Heuristic motivation for approximation (\ref{['appr']})
  16. ...and 1 more sections

Key Result

Theorem 1

Assume that for all $t\in[H[,$ the mappings $\mathcal{K}_{t}(\cdot,\cdot,\varepsilon)$ for any fixed $\varepsilon\in \mathsf{E},$$R_t$ and $F$ are uniformly bounded and continuous functions on $\mathsf{S}\times \mathsf{A}$ and $\mathsf{S},$ respectively. For any fixed $x\in\mathsf{S},$ it then holds Moreover, the supremum in (eq:bellman-q) is attained at some deterministic optimal action $a^{\bold

Figures (1)

  • Figure 1: Results for different values of the parameter $\lambda$ and $F(x)=\log((1+|x|)^2/2)$.

Theorems & Definitions (8)

  • Theorem 1
  • Theorem 2
  • proof
  • Theorem 3
  • Proposition 4
  • Corollary 5
  • Proposition 6
  • proof