What Makes for a Good Saliency Map? Comparing Strategies for Evaluating Saliency Maps in Explainable AI (XAI)
Felix Kares, Timo Speith, Hanwei Zhang, Markus Langer
TL;DR
This paper tackles the problem of how to evaluate saliency maps for explainable AI by comparing three popular approaches—Grad-CAM, Guided Backpropagation, and LIME—across three evaluation families: subjective user measures, objective user abilities, and mathematical metrics. Using a between-subjects online experiment with a sizable sample, it reveals that the different evaluation methods do not agree on which map is best: Grad-CAM most improves user abilities, GBP scores highest on mathematical metrics, and trust/satisfaction show little differentiation. The findings highlight a complex, often counterintuitive relationship between quantitative metrics and human understanding, suggesting that mathematical metrics cannot replace user studies but can complement them. Practically, the work advises context-dependent map selection and advocates a holistic evaluation framework to better capture the multifaceted goals of XAI explanations.
Abstract
Saliency maps are a popular approach for explaining classifications of (convolutional) neural networks. However, it remains an open question as to how best to evaluate salience maps, with three families of evaluation methods commonly being used: subjective user measures, objective user measures, and mathematical metrics. We examine three of the most popular saliency map approaches (viz., LIME, Grad-CAM, and Guided Backpropagation) in a between subject study (N=166) across these families of evaluation methods. We test 1) for subjective measures, if the maps differ with respect to user trust and satisfaction; 2) for objective measures, if the maps increase users' abilities and thus understanding of a model; 3) for mathematical metrics, which map achieves the best ratings across metrics; and 4) whether the mathematical metrics can be associated with objective user measures. To our knowledge, our study is the first to compare several salience maps across all these evaluation methods$-$with the finding that they do not agree in their assessment (i.e., there was no difference concerning trust and satisfaction, Grad-CAM improved users' abilities best, and Guided Backpropagation had the most favorable mathematical metrics). Additionally, we show that some mathematical metrics were associated with user understanding, although this relationship was often counterintuitive. We discuss these findings in light of general debates concerning the complementary use of user studies and mathematical metrics in the evaluation of explainable AI (XAI) approaches.