Know your Trajectory -- Trustworthy Reinforcement Learning deployment through Importance-Based Trajectory Analysis

TL;DR

The paper tackles the challenge of explaining long-horizon RL behavior by introducing a trajectory-level explanation framework that augments Q-based state importance with a radical term encoding goal affinity. It defines I(s,a)= Delta Q(s) × R(s,a) and aggregates to trajectory importance I_tau to rank and select representative trajectories, followed by counterfactual generation for contrastive explanations. Empirical results in Acrobot-v1 and LunarLander-v2 show that the V-Goal metric reliably identifies superior trajectories and yields counterfactuals that justify the chosen path, outperforming the classic Delta Q baseline. This work advances trustworthy RL deployment by enabling high-level, interpretable narratives of agent strategy through trajectory-level analysis and explanation.

Abstract

As Reinforcement Learning (RL) agents are increasingly deployed in real-world applications, ensuring their behavior is transparent and trustworthy is paramount. A key component of trust is explainability, yet much of the work in Explainable RL (XRL) focuses on local, single-step decisions. This paper addresses the critical need for explaining an agent's long-term behavior through trajectory-level analysis. We introduce a novel framework that ranks entire trajectories by defining and aggregating a new state-importance metric. This metric combines the classic Q-value difference with a "radical term" that captures the agent's affinity to reach its goal, providing a more nuanced measure of state criticality. We demonstrate that our method successfully identifies optimal trajectories from a heterogeneous collection of agent experiences. Furthermore, by generating counterfactual rollouts from critical states within these trajectories, we show that the agent's chosen path is robustly superior to alternatives, thereby providing a powerful "Why this, and not that?" explanation. Our experiments in standard OpenAI Gym environments validate that our proposed importance metric is more effective at identifying optimal behaviors compared to classic approaches, offering a significant step towards trustworthy autonomous systems.

Figures (4)

• Figure 1: An agent's observed trajectory (black) versus a longer, suboptimal alternative (gray). Our goal is to explain why the black path was chosen by demonstrating its superior importance.
• Figure 2: Acrobot counterfactual trajectory lengths. The red line is the original trajectory's length. (a) For our method, all counterfactuals are longer (worse) than the original. (b) For the classic method, some counterfactuals are shorter (better), indicating it did not select a truly optimal trajectory to explain.
• Figure 3: LunarLander counterfactual trajectory rewards. The red line represents the original trajectory's reward. (a) For our method, all counterfactuals yield lower rewards. (b) For the classic method, some counterfactuals result in higher rewards.
• Figure 4: LunarLander counterfactual trajectory lengths. The red line represents the original trajectory's length. (a) Counterfactuals from our method’s selected trajectory are probabilistically longer. (b) The classic method’s selected trajectory has counterfactuals that are probabilistically shorter.