Increasing Interpretability of Neural Networks By Approximating Human Visual Saliency

TL;DR

A combination strategy of saliency incorporation and active learning is proposed to reduce the human annotation data required by 80% while maintaining the interpretability and performance increase from human saliency.

Abstract

Understanding specifically where a model focuses on within an image is critical for human interpretability of the decision-making process. Deep learning-based solutions are prone to learning coincidental correlations in training datasets, causing over-fitting and reducing the explainability. Recent advances have shown that guiding models to human-defined regions of saliency within individual images significantly increases performance and interpretability. Human-guided models also exhibit greater generalization capabilities, as coincidental dataset features are avoided. Results show that models trained with saliency incorporation display an increase in interpretability of up to 30% over models trained without saliency information. The collection of this saliency information, however, can be costly, laborious and in some cases infeasible. To address this limitation, we propose a combination strategy of saliency incorporation and active learning to reduce the human annotation data required by 80% while maintaining the interpretability and performance increase from human saliency. Extensive experimentation outlines the effectiveness of the proposed approach across five public datasets and six active learning criteria.

Figures (13)

• Figure 1: Overview of proposed approach. A traditional active learning pipeline is described in (a). In (b), this process is augmented to additionally collect saliency information about the images as well as the labels. These saliency annotations are then incorporated into the training process. In the proposed method, Saliency in Active Learning (SAL), human annotations are initially collected for a small number of iterations of active learning as in (b). After this, all future saliency annotation is delegated to an AI model specialized to produce high fidelity saliency maps (c), thus reducing overall human effort.
• Figure 2: Two examples of model saliency for each of the five studied datasets. In each case, all three models (B1, B2, SAL) classified the image correctly. The DICE score with the ground truth is presented in the upper left corner of each heatmap. Models trained without saliency (B1) focus more on background and spurious features for classification. The model saliency of B2 and SAL are similar, with SAL only requiring 20% of the amount of human saliency.
• Figure 3: Learning curves comparing the overlap of model saliency trained under various scenarios with the ground truth on the test set for five datasets. In all cases the AL criteria was margin uncertainty. Plots (a)-(e) are ResNet50-based, while (f) shows the use of SAL with SwinTransformer. Aligning with the research questions in the introduction, results show the following: 1) models trained with saliency incorporated (B2/SAL) have significantly higher overlap/interpretability than those trained without (B1), 2) SAL effectively replicates the performance of B2 with 80% fewer human annotations, and 3) the same trends can be seen across all five datasets. Each learning curve shows the mean of 8 AL runs, with the shaded area representing $\pm1\sigma$.
• Figure 4: Plot details the classification performance over the interpretability. The shapes represent the training approach used, and the colors represent the active learning selection criteria used. Results show: 1) SAL ($\mathord{\text{✚}}$) matches the classification performance and interpretability of models trained with full saliency ($\blacklozenge$) across all AL criteria, and 2) SAL increases performance over TAIT baseline ($\blacktriangledown$), showing the effectiveness of combining saliency incorporation with active learning. Each point is the mean of 8 independent runs.
• Figure 5: Learning curves comparing the accuracy of model saliency trained under various scenarios on the test set for five different datasets. In all cases the AL criteria was margin uncertainty. Plots (a)-(e) are ResNet50-based, while (f) shows the use of SAL with SwinTransformer. Each learning curve shows the mean of 8 AL runs, with the shaded area representing $\pm1\sigma$.