Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

STACHE: Local Black-Box Explanations for Reinforcement Learning Policies

Andrew Elashkin, Orna Grumberg

TL;DR

STACHE tackles the challenge of local, black-box explanations for discrete RL policies by defining Robustness Regions and Minimal Counterfactuals within a factored state space. It offers an exact, model-agnostic BFS-based method to compute these explanations without surrogate models, ensuring fidelity to the agent's actual behavior. Empirical results in Taxi-v3 and MiniGrid show how policy logic crystallizes during training, with brittle, high-precision actions developing narrow stability regions and more stable navigation widening those regions. The framework provides actionable debugging insights for reliability and safety in real-world RL deployments.

Abstract

Reinforcement learning agents often behave unexpectedly in sparse-reward or safety-critical environments, creating a strong need for reliable debugging and verification tools. In this paper, we propose STACHE, a comprehensive framework for generating local, black-box explanations for an agent's specific action within discrete Markov games. Our method produces a Composite Explanation consisting of two complementary components: (1) a Robustness Region, the connected neighborhood of states where the agent's action remains invariant, and (2) Minimal Counterfactuals, the smallest state perturbations required to alter that decision. By exploiting the structure of factored state spaces, we introduce an exact, search-based algorithm that circumvents the fidelity gaps of surrogate models. Empirical validation on Gymnasium environments demonstrates that our framework not only explains policy actions, but also effectively captures the evolution of policy logic during training - from erratic, unstable behavior to optimized, robust strategies - providing actionable insights into agent sensitivity and decision boundaries.

STACHE: Local Black-Box Explanations for Reinforcement Learning Policies

TL;DR

STACHE tackles the challenge of local, black-box explanations for discrete RL policies by defining Robustness Regions and Minimal Counterfactuals within a factored state space. It offers an exact, model-agnostic BFS-based method to compute these explanations without surrogate models, ensuring fidelity to the agent's actual behavior. Empirical results in Taxi-v3 and MiniGrid show how policy logic crystallizes during training, with brittle, high-precision actions developing narrow stability regions and more stable navigation widening those regions. The framework provides actionable debugging insights for reliability and safety in real-world RL deployments.

Abstract

Reinforcement learning agents often behave unexpectedly in sparse-reward or safety-critical environments, creating a strong need for reliable debugging and verification tools. In this paper, we propose STACHE, a comprehensive framework for generating local, black-box explanations for an agent's specific action within discrete Markov games. Our method produces a Composite Explanation consisting of two complementary components: (1) a Robustness Region, the connected neighborhood of states where the agent's action remains invariant, and (2) Minimal Counterfactuals, the smallest state perturbations required to alter that decision. By exploiting the structure of factored state spaces, we introduce an exact, search-based algorithm that circumvents the fidelity gaps of surrogate models. Empirical validation on Gymnasium environments demonstrates that our framework not only explains policy actions, but also effectively captures the evolution of policy logic during training - from erratic, unstable behavior to optimized, robust strategies - providing actionable insights into agent sensitivity and decision boundaries.

Paper Structure

This paper contains 27 sections, 2 equations, 3 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Implementation:
  3. Related Work
  4. Explainable AI (XAI).
  5. Explainable RL (XRL).
  6. Robustness in RL.
  7. Preliminaries
  8. User-Defined Factorization of a Markov Game
  9. Hybrid Factored State Distance
  10. Methodology
  11. Robustness Regions
  12. Characterizing the States within Robustness Regions
  13. Minimal Counterfactual State
  14. Composite Explanations
  15. Nature of Explanatory Robustness
  16. ...and 12 more sections

Figures (3)

  • Figure 1: Visualisation of a composite explanation for an optimal policy $\pi_{\text{optimal}}$ in state $s_0=(0,1,2,1)$ (Taxi at (0,1), Passenger at Y, Dest at G). (a) The large robustness region shows the stability of the South action. (b) The minimal counterfactuals show the sensitive decision boundaries.
  • Figure 2: Evolution of Robustness Regions for state $s_1=(0,0,0,2)$ across training stages. Colored cells indicate stability (same action as seed). As the agent learns the highly specific Pickup logic, the robustness region shrinks from a random collection of states (a) to a precise configuration (b, c), demonstrating that for critical actions, high performance correlates with low robustness region size (high specificity).
  • Figure 3: Diagnosing policy brittleness in MiniGrid state $s_{38}$. (a) Robustness region map when the agent faces Down. Blue arrow indicates initial position. (b) Robustness region map for the same state when the agent faces Right. Robustness region is extremely small, indicating that the action Move Forward is highly unstable at this location. (c) A minimal counterfactual illustrating that simply moving the agent one cell up causes the action to flip to Right, revealing a brittle change in strategy.

Theorems & Definitions (5)

  • Definition 3.1: Markov Game
  • Definition 3.2: User-Defined Factorization $\mathcal{F}$
  • Definition 3.3: Hybrid Factored State Distance
  • Definition 4.1: Robustness Region
  • Definition 4.2: Minimal Counterfactual State