Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Formal Quality Measures for Predictors in Markov Decision Processes

Christel Baier, Sascha Klüppelholz, Jakob Piribauer, Robin Ziemek

TL;DR

This paper addresses how to quantify the quality of predictors for undesired events in systems modeled as Markov decision processes. It introduces two complementary approaches: (i) average-case quality measures obtained by uniformly sampling memoryless randomized policies ($MR$) and evaluating classifier-like metrics such as precision, recall, and $F$-score, and (ii) a causal-volume viewpoint using probability-raising (PR) policies to quantify how often a predictor causally raises the probability of the event via SPR and GPR notions. The authors formalize the MR policy space as a product of simplices, implement two-copy constructions to express confusion-matrix entries, and enable Monte Carlo or linear/quadratic programming techniques to compute averages and decision problems. They establish complexity results (SPR in P, GPR in NP) and provide practical guidance for estimating averages and causal volumes, illustrating the framework with network-like MDPs. Overall, the work offers a rigorous, dual-angle framework for evaluating predictors in probabilistic, non-deterministic control settings with potential impact on the reliability of adaptive AI systems.

Abstract

In adaptive systems, predictors are used to anticipate changes in the systems state or behavior that may require system adaption, e.g., changing its configuration or adjusting resource allocation. Therefore, the quality of predictors is crucial for the overall reliability and performance of the system under control. This paper studies predictors in systems exhibiting probabilistic and non-deterministic behavior modelled as Markov decision processes (MDPs). Main contributions are the introduction of quantitative notions that measure the effectiveness of predictors in terms of their average capability to predict the occurrence of failures or other undesired system behaviors. The average is taken over all memoryless policies. We study two classes of such notions. One class is inspired by concepts that have been introduced in statistical analysis to explain the impact of features on the decisions of binary classifiers (such as precision, recall, f-score). Second, we study a measure that borrows ideas from recent work on probability-raising causality in MDPs and determines the quality of a predictor by the fraction of memoryless policies under which (the set of states in) the predictor is a probability-raising cause for the considered failure scenario.

Formal Quality Measures for Predictors in Markov Decision Processes

TL;DR

This paper addresses how to quantify the quality of predictors for undesired events in systems modeled as Markov decision processes. It introduces two complementary approaches: (i) average-case quality measures obtained by uniformly sampling memoryless randomized policies () and evaluating classifier-like metrics such as precision, recall, and -score, and (ii) a causal-volume viewpoint using probability-raising (PR) policies to quantify how often a predictor causally raises the probability of the event via SPR and GPR notions. The authors formalize the MR policy space as a product of simplices, implement two-copy constructions to express confusion-matrix entries, and enable Monte Carlo or linear/quadratic programming techniques to compute averages and decision problems. They establish complexity results (SPR in P, GPR in NP) and provide practical guidance for estimating averages and causal volumes, illustrating the framework with network-like MDPs. Overall, the work offers a rigorous, dual-angle framework for evaluating predictors in probabilistic, non-deterministic control settings with potential impact on the reliability of adaptive AI systems.

Abstract

In adaptive systems, predictors are used to anticipate changes in the systems state or behavior that may require system adaption, e.g., changing its configuration or adjusting resource allocation. Therefore, the quality of predictors is crucial for the overall reliability and performance of the system under control. This paper studies predictors in systems exhibiting probabilistic and non-deterministic behavior modelled as Markov decision processes (MDPs). Main contributions are the introduction of quantitative notions that measure the effectiveness of predictors in terms of their average capability to predict the occurrence of failures or other undesired system behaviors. The average is taken over all memoryless policies. We study two classes of such notions. One class is inspired by concepts that have been introduced in statistical analysis to explain the impact of features on the decisions of binary classifiers (such as precision, recall, f-score). Second, we study a measure that borrows ideas from recent work on probability-raising causality in MDPs and determines the quality of a predictor by the fraction of memoryless policies under which (the set of states in) the predictor is a probability-raising cause for the considered failure scenario.

Paper Structure

This paper contains 18 sections, 15 theorems, 48 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Preliminaries
  5. Measuring the Quality of a Predictor
  6. Averaging over Policies
  7. Quality Measures
  8. Practical considerations for averages.
  9. Probability-Raising Policies and the Causal Volume of a Predictor
  10. Causal volumes
  11. Checking Probability-Raising Policies
  12. Checking for SPR Policies
  13. Checking for GPR Policies
  14. Conclusion
  15. Appendix
  16. ...and 3 more sections

Key Result

Lemma 3

For an MR policy $\mathfrak{S}$ of $\mathcal{M}$ also viewed as a policy for $\mathcal{M}_{\mathit{C}}$ the following holds

Figures (3)

  • Figure 1: An experiment where we want to predict $\mathit{Shatter}$
  • Figure 2: An MDP $\mathcal{M}$ depicting a communication network.
  • Figure 3: The confusion matrix for the prediction of $\mathit{E} \subseteq S$ by $\mathit{C} \subseteq S$ for a given policy $\mathfrak{S}$.

Theorems & Definitions (40)

  • Example 1
  • Example 2
  • Lemma 3
  • proof
  • Proposition 4
  • proof
  • Example 5
  • Remark 6
  • Definition 7: Probability-raising policy
  • Example 8
  • ...and 30 more