Gradients of Counterfactuals

TL;DR

The paper tackles the unreliability of raw gradients for attributing feature importance in deep networks due to saturation. It introduces interior gradients and integrated gradients along a counterfactual scaling path, offering an easy, architecture-agnostic attribution method with a solid additivity property. Empirical results on Inception, ligand-based networks, and LSTM language models show interior/integrated gradients better align with human intuition for feature importance and help debugging misclassifications. The approach is computationally efficient and practically useful for widespread adoption, with potential extensions to training scenarios.

Abstract

Gradients have been used to quantify feature importance in machine learning models. Unfortunately, in nonlinear deep networks, not only individual neurons but also the whole network can saturate, and as a result an important input feature can have a tiny gradient. We study various networks, and observe that this phenomena is indeed widespread, across many inputs. We propose to examine interior gradients, which are gradients of counterfactual inputs constructed by scaling down the original input. We apply our method to the GoogleNet architecture for object recognition in images, as well as a ligand-based virtual screening network with categorical features and an LSTM based language model for the Penn Treebank dataset. We visualize how interior gradients better capture feature importance. Furthermore, interior gradients are applicable to a wide variety of deep networks, and have the attribution property that the feature importance scores sum to the the prediction score. Best of all, interior gradients can be computed just as easily as gradients. In contrast, previous methods are complex to implement, which hinders practical adoption.

Figures (14)

• Figure 1: Pixel importance using gradients at the image.
• Figure 2: Saturation in Inception
• Figure 3: Visualization of interior gradients. Notice that the visualizations at lower values of the scaling parameter ($\alpha$) are sharper and much better at surfacing important features of the input image.
• Figure 4: Comparing integrated gradients with gradients at the image. Left-to-right: original input image, label and softmax score for the highest scoring class, visualization of integrated gradients, visualization of gradients at the image. Notice that the visualizations obtained from integrated gradients are better at reflecting distinctive features of the image.
• Figure 5: AOPC (SBMBM15) for integrated gradients and gradients at image.

Theorems & Definitions (1)

• Proposition 1