Explanation Multiplicity in SHAP: Characterization and Assessment

TL;DR

This paper investigates explanation multiplicity in SHAP, showing that multiple internally valid attributions can arise for the same prediction due to both model training randomness and explainer stochasticity. It introduces a dual-seed protocol and three multiplicity settings to disentangle sources, and provides calibrated baselines (Dirichlet for magnitude and Mallows for rankings) to interpret observed disagreement. The authors demonstrate that rank-based stability metrics reveal substantial variability often masked by magnitude-based measures, and that high prediction confidence does not guarantee explanation stability. The work advocates for evaluation practices that align with real-world use, treats explainers as components of decision infrastructure, and offers a framework applicable beyond SHAP to other post-hoc explanations.

Abstract

Post-hoc explanations are widely used to justify, contest, and audit automated decisions in high-stakes domains. SHAP, in particular, is often treated as a reliable account of which features drove an individual prediction. Yet SHAP explanations can vary substantially across repeated runs even when the input, task, and trained model are held fixed. We term this phenomenon explanation multiplicity: multiple internally valid but substantively different explanations for the same decision. We present a methodology to characterize multiplicity in feature-attribution explanations and to disentangle sources due to model training/selection from stochasticity intrinsic to the explanation pipeline. We further show that apparent stability depends on the metric: magnitude-based distances can remain near zero while rank-based measures reveal substantial churn in the identity and ordering of top features. To contextualize observed disagreement, we derive randomized baseline values under plausible null models. Across datasets, model classes, and confidence regimes, we find explanation multiplicity is pervasive and persists even for high-confidence predictions, highlighting the need for metrics and baselines that match the intended use of explanations.

Figures (7)

• Figure 1: Instance-level SHAP explanations for a high-confidence negative prediction vary substantially when only the explainer setting is changed. The resulting top-ranked features differ across explainers, with no overlap in the top three features, potentially undermining explanation-based decision-making. Example based on lending (German Credit), with an FT-Transformer classifier.
• Figure 2: Explanation pipeline and sources of explanation multiplicity. Randomness can enter at (a) model construction (training and selection) and (b) explanation (stochastic SHAP approximation, e.g., background resampling), yielding multiple explanations for the same instance.
• Figure 3: Model vs. explainer multiplicity (dissection). Violin plots decomposed by the source of randomness across three datasets (rows) and three metrics (columns: $\ell_2$, Top-$k$ Jaccard, and RBO). Orange violins vary only the explainer seed with the model fixed; green violins vary the model seed with the explainer fixed. Shaded bands denote randomized baseline ranges. German Credit and Diabetes show predominantly model-induced multiplicity, whereas ACS Income exhibits stronger explainer-induced multiplicity.
• Figure 4: Explainer-seed feature-wise multiplicity. With the model fixed, we vary the explainer seed and report feature-level explanation multiplicity for top features. Bar length shows mean pairwise attribution change; annotations indicate average importance.
• Figure 5: Diabetes and ACS Income overall sensitivity. Violin plots of overall sensitivity across repeated runs for Diabetes (top row) and ACS Income (bottom row), measured by $\ell_2$, Top-$k$ Jaccard, and RBO distances. Shaded bands indicate randomized baseline ranges. FT-Transformer and MLP exhibit Jaccard sensitivity approaching the baseline, whereas $\ell_2$ remains concentrated near zero across models. Notably, ACS Income shows multimodal Jaccard distributions, including a dense low-sensitivity mass and additional modes closer to the baseline.

Theorems & Definitions (2)

• proposition 1: Expected squared $\ell_2$ distance
• definition 1: Mallows model under Kendall-Tau distance mallows1957non