The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations

TL;DR

This work shows that post-hoc counterfactual explanations can be unjustified when they lack connection to training data. It introduces a formal ground-truth justification based on connectedness and a Local Risk Assessment (LRA) to quantify local risk of unjustified counterfactuals. Empirical results across multiple datasets reveal substantial local risk and demonstrate that state-of-the-art post-hoc counterfactual methods often fail to ensure justification, highlighting a gap between interpretability and grounding in data. The findings call for justification-aware explanations and further research into methods that align explanations with ground-truth training data, especially in safety-critical domains.

Abstract

Post-hoc interpretability approaches have been proven to be powerful tools to generate explanations for the predictions made by a trained black-box model. However, they create the risk of having explanations that are a result of some artifacts learned by the model instead of actual knowledge from the data. This paper focuses on the case of counterfactual explanations and asks whether the generated instances can be justified, i.e. continuously connected to some ground-truth data. We evaluate the risk of generating unjustified counterfactual examples by investigating the local neighborhoods of instances whose predictions are to be explained and show that this risk is quite high for several datasets. Furthermore, we show that most state of the art approaches do not differentiate justified from unjustified counterfactual examples, leading to less useful explanations.

Figures (3)

• Figure 1: Two classifiers have been trained on $80\%$ of the dataset (a 2D version of the iris dataset) and have the same accuracy over the test set, $0.78$. Left picture: because of its low robustness, the random forest classifier makes questionable generalizations (e.g. small red square in the dark blue region) Right picture: the support vector classifier makes questionable decisions in regions far away from the training data (red area in the top left corner).
• Figure 2: Illustration of the Local Risk Assessment procedure in the context of binary classification. Left: Definition and Initial Assessment steps; right: Iteration step.
• Figure 3: Illustrative result of the Local Risk Assessment procedure (left: $S_x=1$) and the Vulnerability Evaluation test (right: $J_x=1$) for an instance of the half-moons dataset.

Theorems & Definitions (2)

• Definition 1: Justification
• Definition 2: $\epsilon$-justification