Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Counterfactual Strategies for Markov Decision Processes

Paul Kobialka, Lina Gerlach, Francesco Leofante, Erika Ábrahám, Silvia Lizeth Tapia Tarifa, Einar Broch Johnsen

TL;DR

This work tackles explaining sequential decision-making by introducing counterfactual strategies for Markov Decision Processes (MDPs). It formulates counterfactual strategy synthesis as a non-convex MIQCQP that minimizes a distance between the original and counterfactual strategies while ensuring the desired reduction in reachability probability to a target state, and it extends the framework to generate diverse explanations. The approach enforces actionability by restricting changes to user-controllable actions and demonstrates feasibility on real-world datasets with thousands of states, yielding both single and multiple diverse counterfactual strategies within practical runtimes. The results establish a practical pathway for post-hoc explanations and recourse in complex sequential tasks, with potential impact on processes like loan applications and other sequential decision problems across domains.

Abstract

Counterfactuals are widely used in AI to explain how minimal changes to a model's input can lead to a different output. However, established methods for computing counterfactuals typically focus on one-step decision-making, and are not directly applicable to sequential decision-making tasks. This paper fills this gap by introducing counterfactual strategies for Markov Decision Processes (MDPs). During MDP execution, a strategy decides which of the enabled actions (with known probabilistic effects) to execute next. Given an initial strategy that reaches an undesired outcome with a probability above some limit, we identify minimal changes to the initial strategy to reduce that probability below the limit. We encode such counterfactual strategies as solutions to non-linear optimization problems, and further extend our encoding to synthesize diverse counterfactual strategies. We evaluate our approach on four real-world datasets and demonstrate its practical viability in sophisticated sequential decision-making tasks.

Counterfactual Strategies for Markov Decision Processes

TL;DR

This work tackles explaining sequential decision-making by introducing counterfactual strategies for Markov Decision Processes (MDPs). It formulates counterfactual strategy synthesis as a non-convex MIQCQP that minimizes a distance between the original and counterfactual strategies while ensuring the desired reduction in reachability probability to a target state, and it extends the framework to generate diverse explanations. The approach enforces actionability by restricting changes to user-controllable actions and demonstrates feasibility on real-world datasets with thousands of states, yielding both single and multiple diverse counterfactual strategies within practical runtimes. The results establish a practical pathway for post-hoc explanations and recourse in complex sequential tasks, with potential impact on processes like loan applications and other sequential decision problems across domains.

Abstract

Counterfactuals are widely used in AI to explain how minimal changes to a model's input can lead to a different output. However, established methods for computing counterfactuals typically focus on one-step decision-making, and are not directly applicable to sequential decision-making tasks. This paper fills this gap by introducing counterfactual strategies for Markov Decision Processes (MDPs). During MDP execution, a strategy decides which of the enabled actions (with known probabilistic effects) to execute next. Given an initial strategy that reaches an undesired outcome with a probability above some limit, we identify minimal changes to the initial strategy to reduce that probability below the limit. We encode such counterfactual strategies as solutions to non-linear optimization problems, and further extend our encoding to synthesize diverse counterfactual strategies. We evaluate our approach on four real-world datasets and demonstrate its practical viability in sophisticated sequential decision-making tasks.
Paper Structure (19 sections, 3 theorems, 8 equations, 5 figures, 6 tables)

This paper contains 19 sections, 3 theorems, 8 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Outline.
  3. Preliminaries
  4. Markov Decision Processes
  5. Non-linear Optimization
  6. Counterfactual Explanations
  7. Counterfactual Explanations for MDPs
  8. Computing Counterfactual Strategies
  9. Remark.
  10. Diverse Counterfactual Strategies.
  11. Experimental Evaluation
  12. Experimental Design and Setup
  13. Computing Counterfactual Strategies
  14. Diverse Counterfactuals
  15. Related Work
  16. ...and 4 more sections

Key Result

Lemma 1

Assume a solution to $P$, assigning to each variable $v$ the value $\nu(v)$. Let $\sigma'$ be the strategy for $\mathcal{M}$ with $\sigma'(s)(a)=\nu(p_{sa})$ for all $s\in S$ and $a\in A(s)$. Then the objective function value as specified in Constraint (eq:min) equals $d(\sigma,\sigma')$.

Figures (5)

  • Figure 1: Running example of a loan application procedure.
  • Figure 2: Runtime comparison.
  • Figure 3: Results for diverse counterfactual strategies.
  • Figure 4: Results for diverse counterfactual strategies for each value of $\gamma$.
  • Figure 5: Comparison of two counterfactual strategies for BPIC'12. We only depict states that are close to a state where any counterfactual strategy differs from the initial strategy. Actions are colored blue if the first counterfactual strategy differs from the initial strategy for that action, and are dashed if the second counterfactual strategy differs.

Theorems & Definitions (9)

  • Example 1
  • Definition 1: Counterfactual Strategy
  • Example 2
  • Definition 2: $\epsilon$-Counterfactual Strategy
  • Example 3
  • Lemma 1
  • Theorem 1: Soundness and Completeness
  • Theorem 2
  • proof