Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sample Complexity for Markov Decision Processes and Stochastic Optimal Control with Static Risk Measures

Cristian Chávez, Yan Li

Abstract

We present an elementary state augmentation method for a class of static risk measure applied to the total cost for both Markov decision processes and stochastic optimal control, such that dynamic programming equations can be derived on the augmented space. Through this we discuss the sample complexities of these two problems for both finite-horizon and infinite-horizon settings. We demonstrate the application of the proposed approach through studying distributionally robust functional generated by $φ$-divergences including conditional value-at-risk.

Sample Complexity for Markov Decision Processes and Stochastic Optimal Control with Static Risk Measures

Abstract

We present an elementary state augmentation method for a class of static risk measure applied to the total cost for both Markov decision processes and stochastic optimal control, such that dynamic programming equations can be derived on the augmented space. Through this we discuss the sample complexities of these two problems for both finite-horizon and infinite-horizon settings. We demonstrate the application of the proposed approach through studying distributionally robust functional generated by -divergences including conditional value-at-risk.

Paper Structure

This paper contains 17 sections, 32 theorems, 141 equations.

Table of Contents

  1. Introduction
  2. Risk-averse Markov Decision Processes
  3. Finite-horizon Risk-averse Markov Decision Processes
  4. Sample Complexity of Risk-averse Policy Evaluation and Optimization
  5. Risk-averse Policy Evaluation
  6. Risk-averse Policy Optimization
  7. Infinite-horizon Risk-averse Markov Decision Processes
  8. Sample Complexity for Risk-averse Policy Evaluation and Optimization
  9. Risk-averse Stochastic Optimal Control
  10. Finite-horizon Risk-averse Stochastic Optimal Control
  11. Sample Complexity of Risk-averse Policy Evaluation and Optimization
  12. Risk-averse Policy Evaluation
  13. Risk-averse Policy Optimization
  14. Infinite-horizon Risk-averse Stochastic Optimal Control
  15. Sample Complexity of Risk-averse Policy Evaluation and Optimization
  16. ...and 2 more sections

Key Result

Proposition 2.1

We have for any $\pi \in \Pi(\mathcal{M})$, $\blacktriangleleft$$\blacktriangleleft$

Theorems & Definitions (39)

  • Example 1.1: Conditional value-at-risk
  • Example 1.2: $\phi$-divergence distributionally robust functional
  • Definition 2.1: Finite-horizon augmented MDP
  • Proposition 2.1
  • Proposition 2.2
  • Theorem 2.1
  • Remark 2.1
  • Theorem 2.2
  • Lemma 2.1
  • Theorem 2.3
  • ...and 29 more