Beyond Satisfaction: From Placebic to Actionable Explanations For Enhanced Understandability

TL;DR

This study challenges the reliance on subjective satisfaction as a metric for assessing AI explanations by contrasting actionable and placebic explanations within a Social Security filing-age optimization task. The authors employ a between-subjects design with three protocols (None, Placebic, Actionable) and measure objective performance, learning, and subjective satisfaction. Findings show that actionable explanations improve objective performance and learning, while placebic explanations do not outperform no explanations, even though subjective satisfaction does not reliably reflect these differences. The work argues for evaluation frameworks that combine objective task performance with subjective assessments to accurately gauge explanation quality, with implications for pedagogical and user-centric AI systems.

Abstract

Explainable AI (XAI) presents useful tools to facilitate transparency and trustworthiness in machine learning systems. However, current evaluations of system explainability often rely heavily on subjective user surveys, which may not adequately capture the effectiveness of explanations. This paper critiques the overreliance on user satisfaction metrics and explores whether these can differentiate between meaningful (actionable) and vacuous (placebic) explanations. In experiments involving optimal Social Security filing age selection tasks, participants used one of three protocols: no explanations, placebic explanations, and actionable explanations. Participants who received actionable explanations significantly outperformed the other groups in objective measures of their mental model, but users rated placebic and actionable explanations as equally satisfying. This suggests that subjective surveys alone fail to capture whether explanations truly support users in building useful domain understanding. We propose that future evaluations of agent explanation capabilities should integrate objective task performance metrics alongside subjective assessments to more accurately measure explanation quality. The code for this study can be found at https://github.com/Shymkis/social-security-explainer.

Figures (7)

• Figure 1: User interface in practice section with a placebic explanation.
• Figure 2: Demographic distributions.
• Figure 3: Distributions of both objective performance metrics across protocols.
• Figure 4: Tukey's HSD results for both objective performance metrics.
• Figure 5: Distributions of the three subjective survey metrics across protocols.