Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Risk-averse formulations of Stochastic Optimal Control and Markov Decision Processes

Alexander Shapiro, Yan Li

TL;DR

This paper develops a unified framework for risk-averse and distributionally robust optimization in sequential decision problems, focusing on Stochastic Optimal Control (SOC) and Markov Decision Processes (MDP). It introduces conditional and nested risk functionals, highlights Value-at-Risk ($V@R$), and derives necessary and sufficient conditions for the existence of non-randomized optimal policies, along with finite- and infinite-horizon dynamic programs. It provides VaR-based sample-complexity results, analyzes rectangularity and nested robust formulations, and discusses implications for policy existence and computational tractability. Overall, the work connects risk-averse and distributionally robust viewpoints in multistage settings and offers practical guidance for decision-making under model uncertainty and data limitations.

Abstract

The aim of this paper is to investigate risk-averse and distributionally robust modeling of Stochastic Optimal Control (SOC) and Markov Decision Process (MDP). We discuss construction of conditional nested risk functionals, a particular attention is given to the Value-at-Risk measure. Necessary and sufficient conditions for existence of non-randomized optimal policies in the framework of robust SOC and MDP are derived. We also investigate sample complexity of optimization problems involving the Value-at-Risk measure.

Risk-averse formulations of Stochastic Optimal Control and Markov Decision Processes

TL;DR

This paper develops a unified framework for risk-averse and distributionally robust optimization in sequential decision problems, focusing on Stochastic Optimal Control (SOC) and Markov Decision Processes (MDP). It introduces conditional and nested risk functionals, highlights Value-at-Risk (), and derives necessary and sufficient conditions for the existence of non-randomized optimal policies, along with finite- and infinite-horizon dynamic programs. It provides VaR-based sample-complexity results, analyzes rectangularity and nested robust formulations, and discusses implications for policy existence and computational tractability. Overall, the work connects risk-averse and distributionally robust viewpoints in multistage settings and offers practical guidance for decision-making under model uncertainty and data limitations.

Abstract

The aim of this paper is to investigate risk-averse and distributionally robust modeling of Stochastic Optimal Control (SOC) and Markov Decision Process (MDP). We discuss construction of conditional nested risk functionals, a particular attention is given to the Value-at-Risk measure. Necessary and sufficient conditions for existence of non-randomized optimal policies in the framework of robust SOC and MDP are derived. We also investigate sample complexity of optimization problems involving the Value-at-Risk measure.

Paper Structure

This paper contains 14 sections, 7 theorems, 116 equations.

Table of Contents

  1. Introduction
  2. Risk functionals
  3. Sample complexity
  4. Conditional risk functionals
  5. Multistage setting.
  6. Rectangularity.
  7. Nested functionals.
  8. Risk averse Stochastic Optimal Control
  9. Rectangular setting.
  10. Infinite horizon risk averse SOC.
  11. Sample complexity of infinite horizon risk averse SOC
  12. Randomized policies
  13. Risk averse Markov Decision Processes
  14. Infinite horizon setting

Key Result

Proposition 2.1

Suppose that condition epsilon holds. Then for $\delta>0$ and $N\ge \hbox{\small$\frac{1}{2}$} \kappa^{-2}\log(2/\delta)$ the empirical ${\sf V@R}^{P_N}_{\alpha}$ is equal to ${\sf V@R}^{P}_{\alpha}(Z)$ with probability at least $1-\delta$.

Theorems & Definitions (19)

  • Definition 2.1
  • Example 2.1: Robust Value-at-Risk
  • Proposition 2.1
  • Proposition 2.2
  • proof
  • Theorem 2.1
  • proof
  • Corollary 2.1
  • Definition 2.2: Regular Probability Kernel
  • Remark 2.1
  • ...and 9 more