Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Risk-aware Markov Decision Processes Using Cumulative Prospect Theory

Thomas Brihaye, Krishnendu Chatterjee, Stefanie Mohr, Maximilian Weininger

TL;DR

The paper advances risk-aware sequential decision making by integrating Cumulative Prospect Theory (CPT) into Markov chains and Markov decision processes. It introduces a new, intuitive CPT-value defined on prospects induced by strategies, enabling a direct reduction to multi-objective reachability queries and revealing that memoryless randomized strategies suffice for optimal CPT-value. It proves that CPT-value computation for Markov chains lies in PTIME, while for MDPs it is EXPTIME, with a fixed-parameter tractable option when the number of outcomes, the largest utility, and the precision are fixed. A stopping-MDP reduction and a Pareto-frontier approach underpin the algorithmic framework, including guarantees for approximation accuracy and strategy extraction. The results offer a principled, deterministic route to risk-aware planning under CPT in large sequential settings and open paths to extensions such as mean payoff and dynamic/probabilistic CPT variants.

Abstract

Cumulative prospect theory (CPT) is the first theory for decision-making under uncertainty that combines full theoretical soundness and empirically realistic features [P.P. Wakker - Prospect theory: For risk and ambiguity, Page 2]. While CPT was originally considered in one-shot settings for risk-aware decision-making, we consider CPT in sequential decision-making. The most fundamental and well-studied models for sequential decision-making are Markov chains (MCs), and their generalization Markov decision processes (MDPs). The complexity theoretic study of MCs and MDPs with CPT is a fundamental problem that has not been addressed in the literature. Our contributions are as follows: First, we present an alternative viewpoint for the CPT-value of MCs and MDPs. This allows us to establish a connection with multi-objective reachability analysis and conclude the strategy complexity result that memoryless randomized strategies are necessary and sufficient for optimality. Second, based on this connection, we provide an algorithm for computing the CPT-value in MDPs with infinite-horizon objectives. We show that the problem is in EXPTIME and fixed-parameter tractable. Moreover, we provide a polynomial-time algorithm for the special case of MCs.

Risk-aware Markov Decision Processes Using Cumulative Prospect Theory

TL;DR

The paper advances risk-aware sequential decision making by integrating Cumulative Prospect Theory (CPT) into Markov chains and Markov decision processes. It introduces a new, intuitive CPT-value defined on prospects induced by strategies, enabling a direct reduction to multi-objective reachability queries and revealing that memoryless randomized strategies suffice for optimal CPT-value. It proves that CPT-value computation for Markov chains lies in PTIME, while for MDPs it is EXPTIME, with a fixed-parameter tractable option when the number of outcomes, the largest utility, and the precision are fixed. A stopping-MDP reduction and a Pareto-frontier approach underpin the algorithmic framework, including guarantees for approximation accuracy and strategy extraction. The results offer a principled, deterministic route to risk-aware planning under CPT in large sequential settings and open paths to extensions such as mean payoff and dynamic/probabilistic CPT variants.

Abstract

Cumulative prospect theory (CPT) is the first theory for decision-making under uncertainty that combines full theoretical soundness and empirically realistic features [P.P. Wakker - Prospect theory: For risk and ambiguity, Page 2]. While CPT was originally considered in one-shot settings for risk-aware decision-making, we consider CPT in sequential decision-making. The most fundamental and well-studied models for sequential decision-making are Markov chains (MCs), and their generalization Markov decision processes (MDPs). The complexity theoretic study of MCs and MDPs with CPT is a fundamental problem that has not been addressed in the literature. Our contributions are as follows: First, we present an alternative viewpoint for the CPT-value of MCs and MDPs. This allows us to establish a connection with multi-objective reachability analysis and conclude the strategy complexity result that memoryless randomized strategies are necessary and sufficient for optimality. Second, based on this connection, we provide an algorithm for computing the CPT-value in MDPs with infinite-horizon objectives. We show that the problem is in EXPTIME and fixed-parameter tractable. Moreover, we provide a polynomial-time algorithm for the special case of MCs.
Paper Structure (50 sections, 26 theorems, 62 equations, 10 figures, 3 algorithms)

This paper contains 50 sections, 26 theorems, 62 equations, 10 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Cumulative Prospect Theory
  4. Outcomes and Prospects
  5. Evaluating Prospects Using Expected Utility Theory
  6. Evaluating Prospects Using Cumulative Prospect Theory
  7. Markov Systems and Their Semantics
  8. Markov Systems
  9. Semantics: Paths and Strategies
  10. Objectives
  11. CPT-value and Problem Statement
  12. Expected Value and CPT-value
  13. Problem statement
  14. Technical Notions Required for the Solution
  15. Strongly Connected Components
  16. ...and 35 more sections

Key Result

Lemma 3.0

For every MC $\mathsf{M}$ and WR-objective $\Phi$, it holds that $\mathbb{P}_{\mathsf{M}}[\Phi(\rho)=o_i] = p_i$, where $(\vec{o},\vec{p}) = \mathsf{prospect}(\mathsf{M},\Phi)$.

Figures (10)

  • Figure 1: A depiction of a common utility function $\mathsf{u}$.
  • Figure 2: \ref{['ex:intro-motivate']} as MDP. We omit the self looping actions on states $s_1, \ldots, s_4$.
  • Figure 3: (a) An example MDP. (b) Pareto frontier and achievable points for the multi-objective query $\mathcal{Q} = (\textcolor{blue!90!white}{\{s_1\}},\textcolor{red!50!white}{\{s_3\}})$.
  • Figure 4: A simple example of a non-stopping MDP. Transition probabilities are omitted, as they are equal to 1.
  • Figure 5: (a,b) Standard function for the utility function $\mathsf{u}$, with $\alpha=0.3$ or $\alpha=0.8$, where (a) was also shown in \ref{['fig:2-functions']}. (c) Standard functions for the probability weighting functions $\mathsf{w}^+$ and $\mathsf{w}^-$.
  • ...and 5 more figures

Theorems & Definitions (65)

  • Example 1.1: Motivating example
  • Example 2.1
  • Example 2.2: Extended version in \ref{['app:2-exs']}
  • Remark 1: Exact computation
  • Example 2.3
  • Example 2.4
  • Remark 2: Complexity of Pareto Frontier
  • Lemma 3.0: Correctness of Induced Prospect
  • proof
  • Theorem 3.1: Equivalence of Definitions
  • ...and 55 more