Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Explainable Reinforcement Learning via Temporal Policy Decomposition

Franco Ruggeri, Alessio Russo, Rafia Inam, Karl Henrik Johansson

TL;DR

Temporal Policy Decomposition (TPD) addresses explainability in RL by restoring temporal detail in action outcomes via per-time-step Expected Future Outcomes. It uses off-policy fixed-horizon TD learning to estimate EFOs for all state-action pairs, enabling contrastive explanations between actions. The method supports two outcome classes—rewards and events—and provides a concrete, model-free mechanism to reveal when particular outcomes are likely and how they shape rewards. Experiments in a stochastic Taxi domain demonstrate accurate explanations and show how these insights can guide reward shaping and improve trust in RL systems.

Abstract

We investigate the explainability of Reinforcement Learning (RL) policies from a temporal perspective, focusing on the sequence of future outcomes associated with individual actions. In RL, value functions compress information about rewards collected across multiple trajectories and over an infinite horizon, allowing a compact form of knowledge representation. However, this compression obscures the temporal details inherent in sequential decision-making, presenting a key challenge for interpretability. We present Temporal Policy Decomposition (TPD), a novel explainability approach that explains individual RL actions in terms of their Expected Future Outcome (EFO). These explanations decompose generalized value functions into a sequence of EFOs, one for each time step up to a prediction horizon of interest, revealing insights into when specific outcomes are expected to occur. We leverage fixed-horizon temporal difference learning to devise an off-policy method for learning EFOs for both optimal and suboptimal actions, enabling contrastive explanations consisting of EFOs for different state-action pairs. Our experiments demonstrate that TPD generates accurate explanations that (i) clarify the policy's future strategy and anticipated trajectory for a given action and (ii) improve understanding of the reward composition, facilitating fine-tuning of the reward function to align with human expectations.

Explainable Reinforcement Learning via Temporal Policy Decomposition

TL;DR

Temporal Policy Decomposition (TPD) addresses explainability in RL by restoring temporal detail in action outcomes via per-time-step Expected Future Outcomes. It uses off-policy fixed-horizon TD learning to estimate EFOs for all state-action pairs, enabling contrastive explanations between actions. The method supports two outcome classes—rewards and events—and provides a concrete, model-free mechanism to reveal when particular outcomes are likely and how they shape rewards. Experiments in a stochastic Taxi domain demonstrate accurate explanations and show how these insights can guide reward shaping and improve trust in RL systems.

Abstract

We investigate the explainability of Reinforcement Learning (RL) policies from a temporal perspective, focusing on the sequence of future outcomes associated with individual actions. In RL, value functions compress information about rewards collected across multiple trajectories and over an infinite horizon, allowing a compact form of knowledge representation. However, this compression obscures the temporal details inherent in sequential decision-making, presenting a key challenge for interpretability. We present Temporal Policy Decomposition (TPD), a novel explainability approach that explains individual RL actions in terms of their Expected Future Outcome (EFO). These explanations decompose generalized value functions into a sequence of EFOs, one for each time step up to a prediction horizon of interest, revealing insights into when specific outcomes are expected to occur. We leverage fixed-horizon temporal difference learning to devise an off-policy method for learning EFOs for both optimal and suboptimal actions, enabling contrastive explanations consisting of EFOs for different state-action pairs. Our experiments demonstrate that TPD generates accurate explanations that (i) clarify the policy's future strategy and anticipated trajectory for a given action and (ii) improve understanding of the reward composition, facilitating fine-tuning of the reward function to align with human expectations.
Paper Structure (22 sections, 1 theorem, 23 equations, 4 figures, 3 tables)

This paper contains 22 sections, 1 theorem, 23 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Outline
  4. Related Work
  5. Background
  6. rl
  7. fhtd Learning
  8. Temporal Policy Decomposition
  9. efo
  10. Relevant Classes of efo
  11. Expected Future Rewards
  12. Probabilities of Future Events
  13. Learning efo
  14. Experiments and Results
  15. Experimental Setup
  16. ...and 7 more sections

Key Result

Theorem 3

Assume that Then, under these assumptions and using the update in eq:fhtd-learning, we have that $\lim_{t\to\infty} \hat{Q}_h^{(t+1)}(s,a)= Q_h^\pi(s,a)$ almost surely for every $(s,a)$.

Figures (4)

  • Figure 1: Conceptual overview of the method. Given the sequential interaction between agent and environment, human observers need to interpret each selected action in terms of its expected impact on the future trajectory. Thus, explains the action by generating the for a long-enough prediction horizon.
  • Figure 2: Schematic representation of the training \ref{['fig:block-diagram-training']} and inference \ref{['fig:block-diagram-inference']} phases in . During training, the $Q^{\pi}_{o,h}$ for $h=0,1,\dots,H-1$ using learning . During inference, given a state $s \in \mathcal{S}$ and an action $a \in \mathcal{A}$ to explain, the values $[Q^{\pi}_{o,h}(s,a)]_{h=0}^{H-1}$ are predicted and then decomposed into $[O^{\pi}_h(s,a)]_{h=0}^{H-1}$, which are presented to human users.
  • Figure 3: Explanations for two environment states with different initial fuel levels: $10$\ref{['fig:explanation-1-pair-1-env']} and $9$\ref{['fig:explanation-1-pair-2-env']}. In both scenarios, the action analyzed is the one chosen by the policy. The probabilities of future events \ref{['fig:explanation-1-pair-1-event']}-\ref{['fig:explanation-1-pair-2-event']} reveal the policy’s strategy, while the expected future rewards \ref{['fig:explanation-1-pair-1-reward-component']}-\ref{['fig:explanation-1-pair-2-reward-component']} demonstrate how the probabilities of events map to the expected rewards.
  • Figure 4: Example explanations for an environment state \ref{['fig:explanation-2-env']} where changing the initial action from moving south (optimal action) to moving east (suboptimal action) significantly impacts the future trajectory. The expected return difference \ref{['fig:explanation-2-contrastive-return-component']} highlights why moving south is preferable to moving east. The probabilities of future events \ref{['fig:explanation-2-pair-1-event']}-\ref{['fig:explanation-2-pair-2-event']} and the corresponding expected future rewards \ref{['fig:explanation-2-pair-1-reward-component']}-\ref{['fig:explanation-2-pair-2-reward-component']} illustrate the resulting shift in strategy.

Theorems & Definitions (3)

  • Definition 1: efo
  • Definition 2: fhgvf
  • Theorem 3