Explainable Finite-Memory Policies for Partially Observable Markov Decision Processes

TL;DR

A translation for policies of the finite-state-controller (FSC) form from standard literature is designed and it is shown how the method smoothly generalizes to other variants of finite-memory policies.

Abstract

Partially Observable Markov Decision Processes (POMDPs) are a fundamental framework for decision-making under uncertainty and partial observability. Since in general optimal policies may require infinite memory, they are hard to implement and often render most problems undecidable. Consequently, finite-memory policies are mostly considered instead. However, the algorithms for computing them are typically very complex, and so are the resulting policies. Facing the need for their explainability, we provide a representation of such policies, both (i) in an interpretable formalism and (ii) typically of smaller size, together yielding higher explainability. To that end, we combine models of Mealy machines and decision trees; the latter describing simple, stationary parts of the policies and the former describing how to switch among them. We design a translation for policies of the finite-state-controller (FSC) form from standard literature and show how our method smoothly generalizes to other variants of finite-memory policies. Further, we identify specific properties of recently used "attractor-based" policies, which allow us to construct yet simpler and smaller representations. Finally, we illustrate the higher explainability in a few case studies.

Figures (11)

• Figure 1: The maze example
• Figure 2: A finite state controller with explicit tables. In the tables, an observation is described as a vector of the values of the $7$ observable variables. Transition table $t_0$ below node $n_0$ and $t_1$ below $n_1$. For compactness, we only show posterior observations ($z'$) that are relevant to the transitions.
• Figure 3: Explaining FSCs using DTs. For an observation variable $z$, $z'$ would denote the next observation, i.e., the observation the mouse sees after taking the action. A DT in a node describes a positional policy, a DT below a node describes the next node for the next observation (curved arrows in the FSC indicate possible transitions).
• Figure 4: The refuel model for $6 \times 6$ grid (Left). For initial node, DT policy (Center), DT-FSC transition (Right).
• Figure 5: DT representation of the initial policy for treatment of heart diseases suggested by the FSC. For a test, 0 means not done, 1 is negative, 2 is positive, and 3 is inconclusive.

Theorems & Definitions (12)

• Definition 1: MDP
• Definition 2: POMDP
• Definition 3: Policy
• Definition 4: FSC
• Remark 1
• Definition 5: DT
• Example 1: Maze
• Definition 6: DT-FSC
• Definition 7: skip-FSC
• Theorem 1