Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimal strategies in Markov decision processes with finitely additive evaluations

János Flesch, Arkadi Predtetchinski, William D Sudderth, Xavier Venel

Abstract

We study infinite-horizon Markov decision processes (MDPs) where the decision maker evaluates each of her strategies by aggregating the infinite stream of expected stage-rewards. The crucial feature of our approach is that the aggregation is performed by means of a given diffuse charge (a diffuse finitely additive probability measure) on the set of stages. The results of Neyman [2023] imply that in this setting, in every MDP with finite state and action spaces, the decision maker has a pure optimal strategy as long as the diffuse charge satisfies the time value of money principle. His result raises the question of existence of an optimal strategy without additional assumptions on the aggregation charge. We answer this question in the negative with a counterexample. With a delicately constructed aggregation charge, the MDP has no optimal strategy at all, neither pure nor randomized.

Optimal strategies in Markov decision processes with finitely additive evaluations

Abstract

We study infinite-horizon Markov decision processes (MDPs) where the decision maker evaluates each of her strategies by aggregating the infinite stream of expected stage-rewards. The crucial feature of our approach is that the aggregation is performed by means of a given diffuse charge (a diffuse finitely additive probability measure) on the set of stages. The results of Neyman [2023] imply that in this setting, in every MDP with finite state and action spaces, the decision maker has a pure optimal strategy as long as the diffuse charge satisfies the time value of money principle. His result raises the question of existence of an optimal strategy without additional assumptions on the aggregation charge. We answer this question in the negative with a counterexample. With a delicately constructed aggregation charge, the MDP has no optimal strategy at all, neither pure nor randomized.
Paper Structure (7 sections, 2 theorems, 16 equations)

This paper contains 7 sections, 2 theorems, 16 equations.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. The model
  4. Markov decision processes
  5. Aggregation charge
  6. Non-existence of optimal strategies
  7. Concluding remarks

Key Result

Theorem 2

Consider an MDP with finite state and action spaces. Then, the decision maker hasOne can choose any Blackwell optimal strategy for $\sigma$, i.e., any pure stationary strategy that is optimal for all sufficiently small discount rates. (In the proof of Theorem 6 in Neyman [2023] and the notation the

Theorems & Definitions (5)

  • Example 1
  • Theorem 2: Neyman [2023, Theorem 6]
  • Theorem 3
  • Example 4
  • Example 5