Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Succinct Interaction-Aware Explanations

Sascha Xu, Joscha Cüppers, Jilles Vreeken

TL;DR

This work addresses SHAP's inability to capture feature interactions by proposing iShap, a partition-based extension that identifies interacting feature blocks and explains a model additively over those blocks. It formalizes a regularized objective that balances reconstruction fidelity $\left( f(x) - \sum_{S \in \Pi} v(S) \right)^2$ with a sparsity-like penalty on block size, and uses a statistical pairwise interaction test to prune the search space, guiding the partitioning to meaningful coalitions ${\Pi}^*$. After partitioning, iShap computes Shapley values on the reduced game, yielding succinct yet informative explanations that reveal significant interactions without enumerating all subsets. Empirically, iShap improves interaction discovery, surrogate-model accuracy, and interpretability on synthetic GAMs and real data, including a Covid-19 patient case, while offering scalable exact and greedy variants for practical use.

Abstract

SHAP is a popular approach to explain black-box models by revealing the importance of individual features. As it ignores feature interactions, SHAP explanations can be confusing up to misleading. NSHAP, on the other hand, reports the additive importance for all subsets of features. While this does include all interacting sets of features, it also leads to an exponentially sized, difficult to interpret explanation. In this paper, we propose to combine the best of these two worlds, by partitioning the features into parts that significantly interact, and use these parts to compose a succinct, interpretable, additive explanation. We derive a criterion by which to measure the representativeness of such a partition for a models behavior, traded off against the complexity of the resulting explanation. To efficiently find the best partition out of super-exponentially many, we show how to prune sub-optimal solutions using a statistical test, which not only improves runtime but also helps to detect spurious interactions. Experiments on synthetic and real world data show that our explanations are both more accurate resp. more easily interpretable than those of SHAP and NSHAP.

Succinct Interaction-Aware Explanations

TL;DR

This work addresses SHAP's inability to capture feature interactions by proposing iShap, a partition-based extension that identifies interacting feature blocks and explains a model additively over those blocks. It formalizes a regularized objective that balances reconstruction fidelity with a sparsity-like penalty on block size, and uses a statistical pairwise interaction test to prune the search space, guiding the partitioning to meaningful coalitions . After partitioning, iShap computes Shapley values on the reduced game, yielding succinct yet informative explanations that reveal significant interactions without enumerating all subsets. Empirically, iShap improves interaction discovery, surrogate-model accuracy, and interpretability on synthetic GAMs and real data, including a Covid-19 patient case, while offering scalable exact and greedy variants for practical use.

Abstract

SHAP is a popular approach to explain black-box models by revealing the importance of individual features. As it ignores feature interactions, SHAP explanations can be confusing up to misleading. NSHAP, on the other hand, reports the additive importance for all subsets of features. While this does include all interacting sets of features, it also leads to an exponentially sized, difficult to interpret explanation. In this paper, we propose to combine the best of these two worlds, by partitioning the features into parts that significantly interact, and use these parts to compose a succinct, interpretable, additive explanation. We derive a criterion by which to measure the representativeness of such a partition for a models behavior, traded off against the complexity of the resulting explanation. To efficiently find the best partition out of super-exponentially many, we show how to prune sub-optimal solutions using a statistical test, which not only improves runtime but also helps to detect spurious interactions. Experiments on synthetic and real world data show that our explanations are both more accurate resp. more easily interpretable than those of SHAP and NSHAP.
Paper Structure (36 sections, 5 theorems, 32 equations, 7 figures, 6 tables)

This paper contains 36 sections, 5 theorems, 32 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Theory
  3. Objective
  4. Partitioning
  5. Proof:
  6. Proof:
  7. Explanation
  8. Algorithm
  9. Pairwise Interaction Test
  10. Search
  11. Related Work
  12. Experiments
  13. Discovering Interactions
  14. Surrogate Model Accuracy
  15. Case Study: Covid-19 Patient
  16. ...and 21 more sections

Key Result

Theorem 1

Let $v$ be additive for the variables $x_i$ and $x_j$, so that for all covariates $S \subseteq [d]\setminus \{i,j\}$ there exists a partition $A \cup B = S$ with Then, $x_i$ and $x_j$ do not occur together in the optimal partition ${\Pi}^*$ in regards to Objective eq:objective, i.e.

Figures (7)

  • Figure 1: Comparison of $\textsc{Shap}$ (left), our approach i$\textsc{Shap}$ (middle) and $\textsc{nShap}$ (right) on the Bike Sharing dataset fanaee:12:bikedataset. $\textsc{Shap}$ does not inform about any interactions, while $\textsc{nShap}$ overflows with information. i$\textsc{Shap}$ provides a concise explanation for the high predicted demand: its is a warm and dry day for winter (Season:4, Hum:0.49 and Temp:0.39) and a Saturday with little wind (Weekday:6 and Windspeed:0.15).
  • Figure 2: Interaction graph for predicted survival of a hospitalized COVID-19 patient. The detrimental effect of diabetes and hypertension on survival is alleviated by the relatively young age (55) of the patient.
  • Figure 3: [Higher is better] $\mathit{F1}$ scores of recovered interactions in GAMs. i$\textsc{Shap}$ is more accurate than $\textsc{nShap}$ in detecting full sets of interacting features (left) and pairwise interactions (middle), and can do so on more features. i$\textsc{Shap}$-Greedy is equivalent to i$\textsc{Shap}$-Exact and outperforms $\textsc{nShap}$ across all data settings(right).
  • Figure 4: $R^2$ for surrogate models of i$\textsc{Shap}$, $\textsc{nShap}$, $\textsc{Shap}$ and $\textsc{Lime}$ across different model classes (left) and datasets (middle). i$\textsc{Shap}$ provides the most accurate surrogate model across all model classes and datasets, whilst scaling to more dimension than $\textsc{nShap}$ (right).
  • Figure 5: Comparison between ground truth feature importance (gradient times feature values) and the respective i$\textsc{Shap}$ values and $\textsc{Shap}$ values. On the left, we show the average correlation over all models. In the middle, we show the correlation between the i$\textsc{Shap}$ values and the ground truth for a specific model, on the right, we show the correlation between the ground truth and the $\textsc{Shap}$ values.
  • ...and 2 more figures

Theorems & Definitions (6)

  • Definition 1
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 3
  • Theorem 3