Experiential Explanations for Reinforcement Learning

TL;DR

Reinforcement learning agents can be opaque, hindering human understanding and intervention. The paper introduces Experiential Explanations, a post-hoc method that trains influence predictors alongside the policy to reveal how environmental rewards influence decisions via counterfactual trajectories, without altering the agent. Explanations come in aggregated and local forms and leverage an enhanced reward signal to embed natural language cues, enabling human-friendly, faithful interpretations. Two human studies in MiniGrid and Crafter demonstrate that these explanations improve users' ability to predict agent actions and are perceived as more understandable, useful, and complete than baselines, supporting their potential to enhance human-AI collaboration in sequential decision problems.

Abstract

Reinforcement learning (RL) systems can be complex and non-interpretable, making it challenging for non-AI experts to understand or intervene in their decisions. This is due in part to the sequential nature of RL in which actions are chosen because of their likelihood of obtaining future rewards. However, RL agents discard the qualitative features of their training, making it difficult to recover user-understandable information for "why" an action is chosen. We propose a technique Experiential Explanations to generate counterfactual explanations by training influence predictors along with the RL policy. Influence predictors are models that learn how different sources of reward affect the agent in different states, thus restoring information about how the policy reflects the environment. Two human evaluation studies revealed that participants presented with Experiential Explanations were better able to correctly guess what an agent would do than those presented with other standard types of explanation. Participants also found that Experiential Explanations are more understandable, satisfying, complete, useful, and accurate. Qualitative analysis provides information on the factors of Experiential Explanations that are most useful and the desired characteristics that participants seek from the explanations.

Figures (28)

• Figure 1: In an interaction between the user and the agent, the user expected the agent to go up, but the agent went down instead. The user asks the system for an explanation. The figure shows two types of explanations the user can receive. (A) is an explanation that can be derived directly from the agent's policy. (B) is an explanation that can be derived from the agent's policy with the assistance of additional models called negative and positive influence predictors.
• Figure 2: This figure shows an example of the agent's learned maximum learned utility value at each state and the two influence predictors values visualized on heatmaps. The blue squares are the source of negative rewards, whereas the green square is the source of positive rewards. The purple and yellow shapes have no effect. Intuitively, one can see that the agent's policy (right) is closely associated with the positive influence values and the opposite of the negative influence values.
• Figure 3: The three episodes from the human study, in order presented to participants.
• Figure 4: Summary tables of the codes used to understand how users across the explanation groups precieved the explanations in terms of usefulness and format.
• Figure 5: Summary table of the codes used to understand how users across the explanation groups utilized information from the explanations.