A Comparative User Evaluation of XRL Explanations using Goal Identification

TL;DR

This paper tackles the challenge of evaluating explainable reinforcement learning explanations by testing whether users can infer an agent's long-term goal from its explanations. It introduces a goal-identification methodology that creates multiple agents with distinct reward functions in a single environment (Ms. Pacman) and asks users to predict which goal the agent pursues from explanation content. It compares four explanation mechanisms—Dataset Similarity Explanation (DSE), TRD Summarisation, Specific and Relevant Feature Attribution (SARFA), and Optimal Action Description (OAD)—and finds that only DSE consistently exceeds the random baseline of 25%, with other methods performing near random. The study reveals that users are often overconfident and that self-reported ease or understanding do not reliably predict accuracy, highlighting a gap between perceived and actual actionability. The authors discuss limitations and propose concrete improvements for future evaluation (more questions per participant, attention checks, longer training) to advance XRL explainability toward real-world utility.

Abstract

Debugging is a core application of explainable reinforcement learning (XRL) algorithms; however, limited comparative evaluations have been conducted to understand their relative performance. We propose a novel evaluation methodology to test whether users can identify an agent's goal from an explanation of its decision-making. Utilising the Atari's Ms. Pacman environment and four XRL algorithms, we find that only one achieved greater than random accuracy for the tested goals and that users were generally overconfident in their selections. Further, we find that users' self-reported ease of identification and understanding for every explanation did not correlate with their accuracy.

Figures (10)

• Figure 1: Flowchart of the Goal Identification Task
• Figure 2: Ms Pacman with the different reward sources labelled
• Figure 3: Flowchart of the user survey. Blue boxes represent user questions, and the orange boxes represent information given to the user.
• Figure 4: The number of users who got each possible number of correct answers for each explanation mechanism.
• Figure 5: A bar chart for each explanation mechanism with the accuracy of the explanation mechanism across each agent goal. Each bar's accuracy is averaged across 100 answers from users. The dotted line is 25%, the expected accuracy for random guesses.