Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Blackwell optimality in risk-sensitive stochastic control

Marcin Pitera, Łukasz Stettner

Abstract

In this paper, we consider a discrete-time Markov Decision Process (MDP) on a finite state-action space with a long-run risk-sensitive criterion used as the objective function. We discuss the concept of Blackwell optimality and comment on intricacies which arise when the risk-neutral expectation is replaced by the risk-sensitive entropy. Also, we show the relation between the Blackwell optimality and ultimate stationarity and provide an illustrative example that helps to better understand the structural difference between these two concepts.

Blackwell optimality in risk-sensitive stochastic control

Abstract

In this paper, we consider a discrete-time Markov Decision Process (MDP) on a finite state-action space with a long-run risk-sensitive criterion used as the objective function. We discuss the concept of Blackwell optimality and comment on intricacies which arise when the risk-neutral expectation is replaced by the risk-sensitive entropy. Also, we show the relation between the Blackwell optimality and ultimate stationarity and provide an illustrative example that helps to better understand the structural difference between these two concepts.
Paper Structure (8 sections, 4 theorems, 16 equations, 2 figures)

This paper contains 8 sections, 4 theorems, 16 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Long-run risk-sensitive stochastic control for MDPs
  3. Blackwell optimality
  4. Risk-neutral setup
  5. Risk-sensitive setup
  6. Structural difference between risk-neutral and risk-sensitive setup
  7. Interaction between blackwell optimality and ultimate stationarity in the risk-sensitive setup
  8. Illustrative example

Key Result

Theorem 3.1

Assume C and fix $\gamma=0$. Then, there exists a stationary Markov policy $\pi\in\Pi'$ that satisfies the Blackwell property. Furthermore, policy $\pi$ is optimal for the risk-neutral averaged problem.

Figures (2)

  • Figure 1: The plot presents the optimal policy switching point $n(\beta,\gamma)$ from Markov policy $u_0$ to Markov policy $u_1$ for different values of $\gamma\in [-5,5]$ and $\beta\in (0.5,1)$.
  • Figure 2: The plot presents the difference between $J_{\gamma}(1,u_1;1)$ and $J_{\gamma}(1,u_0;1)$ for different values of $\gamma\in [-5,5]$. Positive difference indicates optimality $u_1$, while negative difference indicates optimality of $u_0$.

Theorems & Definitions (5)

  • Theorem 3.1
  • Theorem 3.2
  • Theorem 3.3
  • Theorem 5.1
  • Remark 5.2