Sanity Checks for Explanation Uncertainty

TL;DR

This paper proposes sanity checks for uncertainty explanation methods, where a weight and data randomization tests are defined for explanations with uncertainty, allowing for quick tests to combinations of uncertainty and explanation methods.

Abstract

Explanations for machine learning models can be hard to interpret or be wrong. Combining an explanation method with an uncertainty estimation method produces explanation uncertainty. Evaluating explanation uncertainty is difficult. In this paper we propose sanity checks for uncertainty explanation methods, where a weight and data randomization tests are defined for explanations with uncertainty, allowing for quick tests to combinations of uncertainty and explanation methods. We experimentally show the validity and effectiveness of these tests on the CIFAR10 and California Housing datasets, noting that Ensembles seem to consistently pass both tests with Guided Backpropagation, Integrated Gradients, and LIME explanations.

Figures (7)

• Figure 1: Example explanation with uncertainty on a FER dataset image using Guided Backpropagation and Ensembles.
• Figure 2: SSIM values for the weight randomization test on CIFAR 10 using Dropout. These values are computed between the explanation representation generated with no weight randomization and with incremental weight randomization. As the amount of weight randomization increases, the similarity between explanation representations decreases.
• Figure 3: Visualization of weight randomization effect on the explanations obtained using Dropout and Integrated Gradients on a CIFAR10 sample. Note how the both mean and standard deviation explanation become almost blank with increasing random weights.
• Figure 4: Visualization of weight randomization effect on the explanations obtained using Dropout and Guided Backpropagation on a CIFAR10 sample. Note how the mean and standard deviation explanation become noisy with increasing random weights.
• Figure 5: Visualization of data randomization effect on explanations obtained using Dropout and GBP and IG. Note how the explanation for the model with random labels is considerably more noisy than explanation from a model with true labels.