Counterfactual Explanations for Continuous Action Reinforcement Learning

TL;DR

This work addresses interpretability in continuous-action reinforcement learning by proposing a counterfactual explanation framework that discovers alternative action sequences yielding higher rewards with minimal deviation from observed trajectories. It extends TD3 with counterfactual trajectory generation and sparse reward shaping, including a constrained-state variant through an augmented MDP formulation. The approach solves two problems, p1 and p2, via a TD3-based optimization and augmented-MDP reduction, and demonstrates strong gains in Diabetes Control and Lunar Lander compared to a naive baseline. Across single- and multi-environment settings, the method shows improved effectiveness, efficiency, and generalization, with the unconstrained variant typically performing best. This work advances trustworthy RL in high-stakes domains by enabling constraint-aware, interpretable counterfactual explanations for continuous actions.

Abstract

Reinforcement Learning (RL) has shown great promise in domains like healthcare and robotics but often struggles with adoption due to its lack of interpretability. Counterfactual explanations, which address "what if" scenarios, provide a promising avenue for understanding RL decisions but remain underexplored for continuous action spaces. We propose a novel approach for generating counterfactual explanations in continuous action RL by computing alternative action sequences that improve outcomes while minimizing deviations from the original sequence. Our approach leverages a distance metric for continuous actions and accounts for constraints such as adhering to predefined policies in specific states. Evaluations in two RL domains, Diabetes Control and Lunar Lander, demonstrate the effectiveness, efficiency, and generalization of our approach, enabling more interpretable and trustworthy RL applications.

Figures (5)

• Figure 1: Observed and counterfactual trajectories of glucose levels (states) and insulin dosages (actions) over one hour.
• Figure 2: Learning curves of Positive Counterfactual Percentage ($\rho_+$) for the Diabetes Control domain.
• Figure 3: Learning curves of Advantage Counterfactual Percentage ($\rho_\mathsf{adv}$) for the Diabetes Control domain.
• Figure 4: Learning curves of Positive Counterfactual Percentage ($\rho_+$) for the Lunar Lander domain.
• Figure 5: Learning curves of Advantage Counterfactual Percentage ($\rho_\mathsf{adv}$) for the Lunar Lander domain.