Explanations are a Means to an End: Decision Theoretic Explanation Evaluation
Ziyang Guo, Berk Ustun, Jessica Hullman
TL;DR
The paper introduces a decision-theoretic framework that treats explanations as information signals whose value is measured by the expected improvement they enable on concrete decision tasks, formalizing three estimands: the Theoretic Value of Explanation, Human-Complementary Value, and Behavioral Value.It defines decision problems, signals, and rational benchmarks, derives theoretical upper bounds and decomposition of explanation value, and provides practical estimators that rely on coarsened information models to avoid overfitting and robust problem specification.Through demonstrations in human–AI decision support and mechanistic interpretability, the work shows that explanations can have substantial theoretical value and meaningful human-complementary effects, though behavioral gains are task-dependent and may be limited in some settings.The framework offers a principled pathway to validate and compare explanation methods using task-aligned metrics, guiding deployment and further research toward actionable, performance-enhancing explanations.
Abstract
Explanations of model behavior are commonly evaluated via proxy properties weakly tied to the purposes explanations serve in practice. We contribute a decision theoretic framework that treats explanations as information signals valued by the expected improvement they enable on a specified decision task. This approach yields three distinct estimands: 1) a theoretical benchmark that upperbounds achievable performance by any agent with the explanation, 2) a human-complementary value that quantifies the theoretically attainable value that is not already captured by a baseline human decision policy, and 3) a behavioral value representing the causal effect of providing the explanation to human decision-makers. We instantiate these definitions in a practical validation workflow, and apply them to assess explanation potential and interpret behavioral effects in human-AI decision support and mechanistic interpretability.