Sparse Explanations of Neural Networks Using Pruned Layer-Wise Relevance Propagation

TL;DR

The paper tackles the challenge of noisy, high-dimensional explanations produced by deep networks by introducing Pruned Layer-Wise Relevance Propagation (PLRP), which sparsifies input- and layer-level relevances while conserving total relevance. PLRP prunes relevance mass per layer based on a threshold tied to a chosen sparsity level $p_l$, and redistributes the pruned mass either by scaling the remaining relevance (PLRP-\$\lambda\$) or by using a modified propagation matrix (PLRP-\$M\$). Evaluations on images (ImageNet/ECSSD with VGG-16/ResNet-50) and genomics (synthetic DNA sequences) show that PLRP yields sparser explanations with higher concentration of relevance in ground-truth regions and only minor losses in faithfulness, with PLRP-\$\lambda\$ often outperforming PLRP-\$M\$. The results suggest improved interpretability and robustness of explanations, and the method can be adapted to other local explanation frameworks beyond LRP. Overall, PLRP provides a practical, input-specific mechanism to obtain clearer, more trustworthy explanations for complex data domains like genomics and high-resolution images.

Abstract

Explainability is a key component in many applications involving deep neural networks (DNNs). However, current explanation methods for DNNs commonly leave it to the human observer to distinguish relevant explanations from spurious noise. This is not feasible anymore when going from easily human-accessible data such as images to more complex data such as genome sequences. To facilitate the accessibility of DNN outputs from such complex data and to increase explainability, we present a modification of the widely used explanation method layer-wise relevance propagation. Our approach enforces sparsity directly by pruning the relevance propagation for the different layers. Thereby, we achieve sparser relevance attributions for the input features as well as for the intermediate layers. As the relevance propagation is input-specific, we aim to prune the relevance propagation rather than the underlying model architecture. This allows to prune different neurons for different inputs and hence, might be more appropriate to the local nature of explanation methods. To demonstrate the efficacy of our method, we evaluate it on two types of data: images and genome sequences. We show that our modification indeed leads to noise reduction and concentrates relevance on the most important features compared to the baseline.

Figures (8)

• Figure 1: Illustration of sparsified explanations using our pruned layer-wise relevance propagation (PLRP): (a) Heatmaps of explanations for image classifications obtained through LRP (left) and their corresponding sparsified versions through PLRP (right). These sparser relevance attributions are more distinct explanations of the images and can help to identify and interpret the most important features. (b) Corresponding (exemplary) DNNs show how neurons with low relevance are removed, leading to sparser relevance attribution in every layer and fewer paths through which relevance is propagated. These sparser relevance attributions in the intermediate layers potentially allow to better understand latent factors of the model.
• Figure 2: Results for ECSSD for metrics sparsity, localization, and robustness for different proportions of pruned relevance mass $p$ ranging from 0 to 0.95 in increments of 0.05 for models VGG16 and ResNet50. For sparsity and localization, the y-axis covers the whole output domain of $[0,1]$. The zoom plots focus on the actual covered output domain for better comparison of the methods.
• Figure 3: Results for ECSSD for faithfulness for different proportions of pruned relevance mass $p$ ranging from 0 to 0.95 in increments of 0.05 (left) and for an exemplary parameterization $p=0.15$ (right) for models VGG16 and ResNet50. The AUC differs only slightly compared to the LRP baseline (left). For higher $p$, the difference becomes larger as more features are pruned. For the exemplary parameterization, the prediction score $f_{c^*}(\boldsymbol{x})$ drops similarly steeply as for the LRP baseline for the features with the highest relevance that are perturbed first (right). The difference in AUC is driven by the less important features that are perturbed later.
• Figure 4: Illustration of relevance attribution via heatmaps for LRP and PLRP with different parameterizations for the VGG16 model. Red pixels indicate positive relevance, while blue are negative. PLRP-$\lambda$ and PLRP-$M$ produce sparser explanations than the baseline LRP, thereby concentrating relevance within the ground truth mask. For the same pruning parameter $p$, PLRP$\lambda$ prunes more features than PLRP-$M$. For higher $p$, both approaches prune relevance within the ground truth mask.
• Figure 5: Illustration of pruned explanations: Logo plots of (a) ground truth mask, (b) relevance scores from LRP, and (c) PLRP-$\lambda$ with $p=0.25$ for the model’s prediction. PLRP reduces noise and creates a sparser explanation compared to the LRP baseline. Thereby, PLRP maintains features that lie within the ground truth even if they have less relevance than the noise at other irrelevant features. Hence, it indeed differs from only applying a threshold.