Not All Explanations are Created Equal: Investigating the Pitfalls of Current XAI Evaluation

TL;DR

Not All Explanations are Created Equal examines how XAI explanations are evaluated and argues that many explanations can inflate user satisfaction regardless of quality (placebic explanations). Using an agent that teaches chess tactics, the study compares none, placebic, and actionable explanations across practice and testing puzzles, measuring both performance and subjective satisfaction. Results show actionable explanations enhance objective puzzle performance and learning, whereas placebic explanations do not consistently improve performance and may impair transfer, while surveys fail to reliably reflect explanation quality. The work highlights the need for rigorous, domain-aware evaluation methods in XAI and clarifies when placebic explanations can mislead assessments, urging a shift toward actionable, learning-oriented explanations.

Abstract

Explainable Artificial Intelligence (XAI) aims to create transparency in modern AI models by offering explanations of the models to human users. There are many ways in which researchers have attempted to evaluate the quality of these XAI models, such as user studies or proposed objective metrics like "fidelity". However, these current XAI evaluation techniques are ad hoc at best and not generalizable. Thus, most studies done within this field conduct simple user surveys to analyze the difference between no explanations and those generated by their proposed solution. We do not find this to provide adequate evidence that the explanations generated are of good quality since we believe any kind of explanation will be "better" in most metrics when compared to none at all. Thus, our study looks to highlight this pitfall: most explanations, regardless of quality or correctness, will increase user satisfaction. We also propose that emphasis should be placed on actionable explanations. We demonstrate the validity of both of our claims using an agent assistant to teach chess concepts to users. The results of this chapter will act as a call to action in the field of XAI for more comprehensive evaluation techniques for future research in order to prove explanation quality beyond user satisfaction. Additionally, we present an analysis of the scenarios in which placebic or actionable explanations would be most useful.

Figures (12)

• Figure 1: Situational preferences for each explanation type.
• Figure 2: Screenshots of the agent interface during the study.
• Figure 3: Experiment design.
• Figure 4: Examples of the tactics featured in the puzzles.
• Figure 5: Demographics data.