Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Blackwell Optimality: Bellman Optimality Is All You Can Get

Victor Boone, Adrienne Tuynman

TL;DR

The work investigates learning higher-order optimalities in Markov decision processes and shows that Bellman optimality governs learnability across all orders, with non-degenerate MDPs (having a unique Bellman optimal policy) being the precisely identifiable class. It introduces HOPE, a consistent algorithm for all orders, and derives a δ-PC stopping rule that ensures finite-time termination for non-degenerate MDPs. A key technical development is the shattering construction, which isolates a Bellman-optimal policy as the unique gain-optimal policy in nearby models, establishing a universal degeneracy collapse across orders. The results unify the understanding of identifiability in high-order RL and point to PAC-based extensions and sample-complexity analyses for near-optimal learning.

Abstract

Although average gain optimality is a commonly adopted performance measure in Markov Decision Processes (MDPs), it is often too asymptotic. Further incorporating measures of immediate losses leads to the hierarchy of bias optimalities, all the way up to Blackwell optimality. In this paper, we investigate the problem of identifying policies of such optimality orders. To that end, for each order, we construct a learning algorithm with vanishing probability of error. Furthermore, we characterize the class of MDPs for which identification algorithms can stop in finite time. That class corresponds to the MDPs with a unique Bellman optimal policy, and does not depend on the optimality order considered. Lastly, we provide a tractable stopping rule that when coupled to our learning algorithm triggers in finite time whenever it is possible to do so.

Towards Blackwell Optimality: Bellman Optimality Is All You Can Get

TL;DR

The work investigates learning higher-order optimalities in Markov decision processes and shows that Bellman optimality governs learnability across all orders, with non-degenerate MDPs (having a unique Bellman optimal policy) being the precisely identifiable class. It introduces HOPE, a consistent algorithm for all orders, and derives a δ-PC stopping rule that ensures finite-time termination for non-degenerate MDPs. A key technical development is the shattering construction, which isolates a Bellman-optimal policy as the unique gain-optimal policy in nearby models, establishing a universal degeneracy collapse across orders. The results unify the understanding of identifiability in high-order RL and point to PAC-based extensions and sample-complexity analyses for near-optimal learning.

Abstract

Although average gain optimality is a commonly adopted performance measure in Markov Decision Processes (MDPs), it is often too asymptotic. Further incorporating measures of immediate losses leads to the hierarchy of bias optimalities, all the way up to Blackwell optimality. In this paper, we investigate the problem of identifying policies of such optimality orders. To that end, for each order, we construct a learning algorithm with vanishing probability of error. Furthermore, we characterize the class of MDPs for which identification algorithms can stop in finite time. That class corresponds to the MDPs with a unique Bellman optimal policy, and does not depend on the optimality order considered. Lastly, we provide a tractable stopping rule that when coupled to our learning algorithm triggers in finite time whenever it is possible to do so.
Paper Structure (40 sections, 33 theorems, 95 equations, 3 figures, 2 algorithms)

This paper contains 40 sections, 33 theorems, 95 equations, 3 figures, 2 algorithms.

Table of Contents

  1. INTRODUCTION
  2. PRELIMINARIES
  3. MDPs and high order optimalities
  4. Interacting with a MDP
  5. Gain, High Order Biases & Optimalities
  6. Fixed-Confidence Identification and delta-Probably Correct Algorithms
  7. Related Works
  8. Contributions
  9. CONSISTENT ALGORITHMS AND MODEL DEGENERACY
  10. Existence of Consistent Algorithms
  11. Dealing with noise in Policy Iteration.
  12. MDPs Learnable in Finite Time
  13. DEGENERATE MODELS AND STOPPING IN FINITE TIME
  14. Non-Degeneracy and Locally Optimal Policies
  15. Making a Bellman Optimal Policy into the Unique Gain Optimal Policy
  16. ...and 25 more sections

Key Result

Theorem 1

For all $n \ge -1$, HOPE$(n)$ (alg:HOPE) is consistent for $\Pi^*_n$.

Figures (3)

  • Figure 1: A class of deterministic transition Markov decision processes parameters by $\theta \in (- \varepsilon, \varepsilon)$. Choices of actions are represented by arrows that represent transitions. Labels are mean rewards. Rewards are reduced Gaussians, i.e., of the form $\mathrm{N}(-, 1)$.
  • Figure 2: Illustration of \ref{['proposition_non_degenerate_general']} --- as long as we can put $\widehat{M}$ and $\widetilde{M}$ arbitrarily close to $M$ but having no common optimal policy, we can never learn an optimal policy for $M$
  • Figure 3: An example of the shattering operation, making a (non-unique) bias optimal policy (in red) into the unique gain optimal policy.

Theorems & Definitions (66)

  • Definition 1
  • Definition 2: Consistency
  • Definition 3: Probable correctedness
  • Theorem 1
  • Definition 4: Non-degenerate models
  • Theorem 2: Characterization of non-degeneracy
  • Corollary 3: Degeneracy collapse
  • Proposition 4
  • proof : Proof Sketch
  • Proposition 5: Shattering Bellman optimality
  • ...and 56 more