Toward Understanding the Disagreement Problem in Neural Network Feature Attribution

TL;DR

This work tackles the disagreement problem in neural network feature attribution by categorizing attribution methods into four groups and using synthetic tabular data to study how distributions and baselines shape explanations. It shows that baseline choices and preprocessing largely drive attribution magnitudes and rank-based disagreements, with reference-based and Shapley-based methods aligning with ground-truth effects under common preprocessing, while gradient-based methods often fail for local decompositions. The study finds that DeepSHAP and DeepLIFT-RC provide robust, interpretable attributions and strong discrimination between important and unimportant features, especially for non-linear relationships. The findings highlight the practical importance of selecting baselines and preprocessing steps to obtain faithful explanations and caution against overreliance on rank-based disagreement metrics.

Abstract

In recent years, neural networks have demonstrated their remarkable ability to discern intricate patterns and relationships from raw data. However, understanding the inner workings of these black box models remains challenging, yet crucial for high-stake decisions. Among the prominent approaches for explaining these black boxes are feature attribution methods, which assign relevance or contribution scores to each input variable for a model prediction. Despite the plethora of proposed techniques, ranging from gradient-based to backpropagation-based methods, a significant debate persists about which method to use. Various evaluation metrics have been proposed to assess the trustworthiness or robustness of their results. However, current research highlights disagreement among state-of-the-art methods in their explanations. Our work addresses this confusion by investigating the explanations' fundamental and distributional behavior. Additionally, through a comprehensive simulation study, we illustrate the impact of common scaling and encoding techniques on the explanation quality, assess their efficacy across different effect sizes, and demonstrate the origin of inconsistency in rank-based evaluation metrics.

Figures (7)

• Figure 1: Pairwise comparison of state-of-the-art feature attribution methods (see Sec. \ref{['Sec:background_methods']} for the method descriptions) on the COMPAS dataset using (left) the mean feature-correlation, (middle) mean instance-wise Kendall rank correlation, and mean rank agreement of the two most important features (details can be found in Appendix \ref{['App:COMPAS']}).
• Figure 2: Resulting attribution values of (a) prediction-sensitive and (b) fixed-reference methods of $1,000$ test instances based on the DGP in Eq. \ref{['Sec_3:dgp']}. The distribution is shown as a violin plot at the top and a bar plot of the same single instance at the bottom.
• Figure 3: Resulting attribution values of (a) reference-based and (b) Shapley-based methods of $1,000$ test instances based on the DGP in Eq. \ref{['Sec_3:dgp']}. The distribution is shown as a violin plot at the top and a bar plot of the same single instance at the bottom.
• Figure 4: Results of the preprocess simulations for state-of-the-art feature attribution methods (y-axis) showing the averaged correlation with the ground-truth effects across $200$ repetitions and $p=12$ features with equal effect strengths. The individual columns represent different types of effects, and the x-axis shows various preprocessing functions. The size-varying dot describes the standard deviation of the aggregation.
• Figure 5: Results of the faithfulness simulations for state-of-the-art feature attribution methods (y-axis) showing the correlation with the ground-truth effects across $200$ repetitions and $p=12$ features with grouped (weak, medium, strong) effect strengths as box plots. The individual columns represent different types of effects, and the x-axis shows the correlation.