Impossibility Theorems for Feature Attribution

TL;DR

This work rigorously analyzes feature attribution methods through a hypothesis-testing lens, showing that for moderately rich models, complete and linear explanations (like SHAP and Integrated Gradients) cannot reliably reveal counterfactual model behaviour and often perform no better than random guessing for end-tasks such as algorithmic recourse and spurious-feature detection. The authors prove general impossibility results under mild assumptions and corroborate them with extensive experiments across tabular and image data, where simpler local methods sometimes outperform the touted complete/linear approaches. A key practical takeaway is that practitioners should explicitly define end-tasks and may resort to brute-force model querying to infer counterfactual behaviour when guarantees are required. The paper also outlines a research direction toward theoretical guarantees for perturbation-based methods and argues for developing methods with explicit performance guarantees tailored to concrete tasks.

Abstract

Despite a sea of interpretability methods that can produce plausible explanations, the field has also empirically seen many failure cases of such methods. In light of these results, it remains unclear for practitioners how to use these methods and choose between them in a principled way. In this paper, we show that for moderately rich model classes (easily satisfied by neural networks), any feature attribution method that is complete and linear -- for example, Integrated Gradients and SHAP -- can provably fail to improve on random guessing for inferring model behaviour. Our results apply to common end-tasks such as characterizing local model behaviour, identifying spurious features, and algorithmic recourse. One takeaway from our work is the importance of concretely defining end-tasks: once such an end-task is defined, a simple and direct approach of repeated model evaluations can outperform many other complex feature attribution methods.

Figures (4)

• Figure 1: Red arrows indicate false implications for complete and linear feature attribution methods, which follows from \ref{['fact:oracle-hypothesis']}. Implication (A) is a standard belief in the literature for feature attribution methods, but we show it is false in general.
• Figure 2: Each line represents a different one-dimensional model. For $x=0.1$ and $\mu=\mathrm{Unif}(-1,1)$, dashed lines receive SHAP$(f,x,\mu)=0$ while solid lines receive SHAP$(f,x,\mu)=1$. The behaviour of models with the same colour is identical within the shaded region, which denotes the neighbourhood $(x-\delta,x+\delta)$ for $\delta=0.2$. Models can behave very differently and all receive the same attribution (e.g., all dashed lines) and models can be identical in a neighbourhood yet receive very different attribution within that neighbourhood (e.g., lines with the same colour).
• Figure 3: Visualizing ROC curves for tabular datasets. A feature attribution method is better for an end-task if the ROC curve is closer to the top left corner on average.
• Figure 4: Visualizing ROC curves for image datasets. A feature attribution method is better for an end-task if the ROC curve is closer to the top left corner on average.

Theorems & Definitions (32)

• Definition 3.1: Complete
• Definition 3.2: Linear
• Theorem 3.3
• Theorem 3.4
• Proposition 3.5
• Proposition 3.6
• Definition 3.7: Recourse
• Definition 3.8: Spurious Features
• Corollary 3.9
• Proposition 3.10