Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reducing Blackwell and Average Optimality to Discounted MDPs via the Blackwell Discount Factor

Julien Grand-Clément, Marek Petrik

TL;DR

The paper introduces the Blackwell discount factor $\gamma_{bw}$, a threshold above which discounted optimal policies are automatically Blackwell- and average-optimal for finite MDPs. It proves existence of $\gamma_{bw}$ and derives a general, instance-dependent upper bound $1-\eta(\mathcal{M})$ using polynomial and algebraic-number separation techniques, enabling a practical reduction of average and Blackwell optimality to discounted optimality without structural assumptions. The work extends these results to robust MDPs, establishing a robust analogue $\gamma_{bw,r}$ for sa-rectangular uncertainty and providing the first algorithms for robust Blackwell-optimal policies via discounted MDP methods. Overall, the results unify stopping criteria under discounting, offering polynomial-time pathways to compute broader notions of optimality through well-understood discounted MDP techniques, with implications for both standard and robust settings.

Abstract

We introduce the Blackwell discount factor for Markov Decision Processes (MDPs). Classical objectives for MDPs include discounted, average, and Blackwell optimality. Many existing approaches to computing average-optimal policies solve for discounted optimal policies with a discount factor close to $1$, but they only work under strong or hard-to-verify assumptions such as ergodicity or weakly communicating MDPs. In this paper, we show that when the discount factor is larger than the Blackwell discount factor $γ_{\mathrm{bw}}$, all discounted optimal policies become Blackwell- and average-optimal, and we derive a general upper bound on $γ_{\mathrm{bw}}$. The upper bound on $γ_{\mathrm{bw}}$ provides the first reduction from average and Blackwell optimality to discounted optimality, without any assumptions, and new polynomial-time algorithms for average- and Blackwell-optimal policies. Our work brings new ideas from the study of polynomials and algebraic numbers to the analysis of MDPs. Our results also apply to robust MDPs, enabling the first algorithms to compute robust Blackwell-optimal policies.

Reducing Blackwell and Average Optimality to Discounted MDPs via the Blackwell Discount Factor

TL;DR

The paper introduces the Blackwell discount factor , a threshold above which discounted optimal policies are automatically Blackwell- and average-optimal for finite MDPs. It proves existence of and derives a general, instance-dependent upper bound using polynomial and algebraic-number separation techniques, enabling a practical reduction of average and Blackwell optimality to discounted optimality without structural assumptions. The work extends these results to robust MDPs, establishing a robust analogue for sa-rectangular uncertainty and providing the first algorithms for robust Blackwell-optimal policies via discounted MDP methods. Overall, the results unify stopping criteria under discounting, offering polynomial-time pathways to compute broader notions of optimality through well-understood discounted MDP techniques, with implications for both standard and robust settings.

Abstract

We introduce the Blackwell discount factor for Markov Decision Processes (MDPs). Classical objectives for MDPs include discounted, average, and Blackwell optimality. Many existing approaches to computing average-optimal policies solve for discounted optimal policies with a discount factor close to , but they only work under strong or hard-to-verify assumptions such as ergodicity or weakly communicating MDPs. In this paper, we show that when the discount factor is larger than the Blackwell discount factor , all discounted optimal policies become Blackwell- and average-optimal, and we derive a general upper bound on . The upper bound on provides the first reduction from average and Blackwell optimality to discounted optimality, without any assumptions, and new polynomial-time algorithms for average- and Blackwell-optimal policies. Our work brings new ideas from the study of polynomials and algebraic numbers to the analysis of MDPs. Our results also apply to robust MDPs, enabling the first algorithms to compute robust Blackwell-optimal policies.
Paper Structure (23 sections, 19 theorems, 44 equations, 2 figures)

This paper contains 23 sections, 19 theorems, 44 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Preliminaries on MDPs
  3. Optimality criteria.
  4. Classical Blackwell optimality
  5. Existing definition and algorithms
  6. Existence of a Blackwell-optimal policy.
  7. Existing algorithms.
  8. Limitations of existing approaches
  9. Blackwell discount factor
  10. Difference from the existing definition.
  11. Reduction to discounted optimality.
  12. Upper bound on $\gamma_{\sf bw}$
  13. Discussion
  14. The case of robust MDPs
  15. Robust Blackwell discount factor.
  16. ...and 8 more sections

Key Result

Theorem 3.2

In any finite MDP, there exists at least one Blackwell-optimal policy: $\Pi^{\star}_{\sf bw} \neq \emptyset$.

Figures (2)

  • Figure 1: MDP instance for Example \ref{['ex:counter-example-0']} (Figure \ref{['fig:example-0']}) . There are three actions in state $0$ and the transitions are deterministic. The instantaneous rewards are represented above the transition arcs. The value functions are represented in Figure \ref{['fig:example-1']}.
  • Figure 2: MDP instance for Example \ref{['ex:multiple-intervals']} (Figure \ref{['fig:example-multiple-intervals']}) and the value functions for $N=5$ (Figure \ref{['fig:value-functions-multiple-intervals']}).

Theorems & Definitions (36)

  • Definition 2.1
  • Definition 3.1
  • Theorem 3.2: blackwell1962discrete
  • Lemma 3.3
  • Corollary 3.4
  • Proposition 3.5
  • Example 3.6
  • Theorem 3.7
  • Example 3.8
  • Definition 4.1
  • ...and 26 more