Evaluating the Ability of Explanations to Disambiguate Models in a Rashomon Set

TL;DR

This work addresses the problem of evaluating explanations in Rashomon sets, where multiple models yield similar predictions but reveal different internal mechanisms. It introduces three guiding principles—Local Contextualization, Model Relativism, and On-manifold Evaluation—and presents AXE, a ground-truth-agnostic explanation evaluation framework that uses per-datapoint top-n feature importance and k-NN surrogates to test explainability. The authors show that traditional ground-truth metrics can force uniform explanations across models, while AXE robustly distinguishes true model differences and detects adversarial fairwashing. The framework enhances reliable model selection and accountability by ensuring explanations reflect genuine model behavior on the data manifold and are resistant to off-manifold manipulation.

Abstract

Explainable artificial intelligence (XAI) is concerned with producing explanations indicating the inner workings of models. For a Rashomon set of similarly performing models, explanations provide a way of disambiguating the behavior of individual models, helping select models for deployment. However explanations themselves can vary depending on the explainer used, and need to be evaluated. In the paper "Evaluating Model Explanations without Ground Truth", we proposed three principles of explanation evaluation and a new method "AXE" to evaluate the quality of feature-importance explanations. We go on to illustrate how evaluation metrics that rely on comparing model explanations against ideal ground truth explanations obscure behavioral differences within a Rashomon set. Explanation evaluation aligned with our proposed principles would highlight these differences instead, helping select models from the Rashomon set. The selection of alternate models from the Rashomon set can maintain identical predictions but mislead explainers into generating false explanations, and mislead evaluation methods into considering the false explanations to be of high quality. AXE, our proposed explanation evaluation method, can detect this adversarial fairwashing of explanations with a 100% success rate. Unlike prior explanation evaluation strategies such as those based on model sensitivity or ground truth comparison, AXE can determine when protected attributes are used to make predictions.

Figures (4)

• Figure 1: Explanation generation: Explainer $\mathcal{E}$ produces an explanation vector $\mathbf{e}$ of signed feature importances using datapoint $\mathbf{x}$, model $m$ and prediction $m(\mathbf{x})$.
• Figure 2: Different Explainers Yield Different Explanations: A neural network predicts diabetes on the "Pima Indians" dataset pima. A single positive (diabetic) prediction is explained using four explainers. Each feature-importance explanation varies, but consists of a signed vector indicating the relative contribution of each input to the model output.
• Figure 3: Explanation evaluation: AXE evaluates the quality $q$ of explanation $\mathbf{e}$ by measuring how accurately prediction $m(\mathbf{x})$ can be recovered from dataset $\mathcal{X}$.
• Figure 4: Explanation Quality is Invariant to Model Selection within the Rashomon Set: Plots showing explanation quality $q$ (color) across $i_1$ and $i_2$ values for explanation $\mathbf{e} = (i_1, i_2)$. Model $m(\mathbf{x})=\beta_0 + \beta_1 X_1 + \beta_2 X_2$ has ground-truth $\mathbf{e}^{*} = (\beta_1, \beta_2) = (0.7,0.3)$. Consider model $m'(\mathbf{x})=\beta'_0 + \beta'_1 X_1 + \beta'_2 X_2$ that makes "mostly" similar predictions and belongs to the same Rashomon set $\{m, m'\} \subseteq \mathcal{M}$. If $\beta'_1 > \beta'_2 > 0$, then these explanation quality measurements remain unchanged (despite changes in ground-truth $\mathbf{e}^{*}$ against which explanation $\mathbf{e}$ will be compared). This violates the model relativism principle.