Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Monte Carlo Grid Dynamic Programming: Almost Sure Convergence and Probability Constraints

Mohammad S. Ramadan, Ahmad Al-Tawaha, Mohamed Shouman, Ahmed Atallah, Ming Jin

TL;DR

The proposed sampling and self-approximating algorithm alleviates the burden of gridding and interpolation traditionally required in DP and demonstrates that the proposed interpolation procedure is well-suited for handling probabilistic constraints by sampling both infeasible and feasible regions.

Abstract

Dynamic Programming (DP) suffers from the well-known ``curse of dimensionality'', further exacerbated by the need to compute expectations over process noise in stochastic models. This paper presents a Monte Carlo-based sampling approach for the state space and an interpolation procedure for the resulting value function, dependent on the process noise density, in a "self-approximating" fashion, eliminating the need for ordering or set-membership tests. We provide proof of almost sure convergence for the value iteration (and consequently, policy iteration) procedure. The proposed meshless sampling and interpolation algorithm alleviates the burden of gridding the state space, traditionally required in DP, and avoids constructing a piecewise constant value function over a grid. Moreover, we demonstrate that the proposed interpolation procedure is well-suited for handling probabilistic constraints by sampling both infeasible and feasible regions. The curse of dimensionality cannot be avoided, however, this approach offers a practical framework for addressing lower-order stochastic nonlinear systems with probabilistic constraints, while eliminating the need for linear interpolations and set membership tests. Numerical examples are presented to further explain and illustrate the convenience of the proposed algorithms.

Monte Carlo Grid Dynamic Programming: Almost Sure Convergence and Probability Constraints

TL;DR

The proposed sampling and self-approximating algorithm alleviates the burden of gridding and interpolation traditionally required in DP and demonstrates that the proposed interpolation procedure is well-suited for handling probabilistic constraints by sampling both infeasible and feasible regions.

Abstract

Dynamic Programming (DP) suffers from the well-known ``curse of dimensionality'', further exacerbated by the need to compute expectations over process noise in stochastic models. This paper presents a Monte Carlo-based sampling approach for the state space and an interpolation procedure for the resulting value function, dependent on the process noise density, in a "self-approximating" fashion, eliminating the need for ordering or set-membership tests. We provide proof of almost sure convergence for the value iteration (and consequently, policy iteration) procedure. The proposed meshless sampling and interpolation algorithm alleviates the burden of gridding the state space, traditionally required in DP, and avoids constructing a piecewise constant value function over a grid. Moreover, we demonstrate that the proposed interpolation procedure is well-suited for handling probabilistic constraints by sampling both infeasible and feasible regions. The curse of dimensionality cannot be avoided, however, this approach offers a practical framework for addressing lower-order stochastic nonlinear systems with probabilistic constraints, while eliminating the need for linear interpolations and set membership tests. Numerical examples are presented to further explain and illustrate the convenience of the proposed algorithms.
Paper Structure (6 sections, 6 theorems, 38 equations, 1 figure, 2 algorithms)

This paper contains 6 sections, 6 theorems, 38 equations, 1 figure, 2 algorithms.

Table of Contents

  1. Introduction
  2. Finite-Horizon
  3. Infinite-Horizon Discounted Cost
  4. Probabilistic Constraints
  5. Numerical Examples
  6. Conclusion

Key Result

Lemma 1

The approximation $\tilde{V}_{k}^{N}$ in (FDPE:approx), when $k=T-1$, converges to $V_{T-1}$ almost surely on $\mathbb X$, as $N \to \infty$.

Figures (1)

  • Figure 1: Example \ref{['example2']}: Colormap of $\kappa$ acquired by Algorithm \ref{['algorithmDPE_infty']} and fitted by a piece-wise surface. White regions correspond to infeasible states.

Theorems & Definitions (15)

  • Remark 1
  • Lemma 1
  • proof
  • Proposition 1
  • proof
  • Corollary 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • ...and 5 more