Evaluating Model Explanations without Ground Truth

TL;DR

The paper addresses the challenge of evaluating local feature-importance explanations without ground-truth explanations by introducing AXE, a ground-truth agnostic, on-manifold evaluation framework. AXE uses per-point, top-n feature importance surrogates via $k$-nearest neighbors to measure how well explanations recover model behavior, satisfying local contextualization, model relativism, and on-manifold evaluation. Through synthetic demonstrations and large-scale experiments, AXE is shown to detect explanation fairwashing, outperform traditional ground-truth and sensitivity metrics in reliability, and remain robust to hyperparameter choices. The approach provides a principled, practical tool for comparing explanations and enhancing trust in XAI methods across datasets and models.

Abstract

There can be many competing and contradictory explanations for a single model prediction, making it difficult to select which one to use. Current explanation evaluation frameworks measure quality by comparing against ideal "ground-truth" explanations, or by verifying model sensitivity to important inputs. We outline the limitations of these approaches, and propose three desirable principles to ground the future development of explanation evaluation strategies for local feature importance explanations. We propose a ground-truth Agnostic eXplanation Evaluation framework (AXE) for evaluating and comparing model explanations that satisfies these principles. Unlike prior approaches, AXE does not require access to ideal ground-truth explanations for comparison, or rely on model sensitivity - providing an independent measure of explanation quality. We verify AXE by comparing with baselines, and show how it can be used to detect explanation fairwashing. Our code is available at https://github.com/KaiRawal/Evaluating-Model-Explanations-without-Ground-Truth.

Figures (7)

• Figure 1: Different Explainers Yield Different Explanations: A neural network predicts diabetes on the "Pima Indians" dataset pima. A single positive (diabetic) prediction is explained using four explainers. These feature-importance explanations, visualized here as "force-plots", consist of a signed vector indicating the relative contribution of each input to the model output. They disagree with each other. Section \ref{['subsec:prior']} details the explainers, and section \ref{['subsec:framework']} evaluates these four explanations using AXE.
• Figure 2: Explanation Generation (a) and Evaluation (b): AXE measures how well a given explanation can help emulate model behavior. See section \ref{['subsec:framework']} for full algorithm.
• Figure 3: Violations of local contextualization and model relativism: Plots showing explanation quality $q$ (color) across $i_1$ and $i_2$ values for explanation $\mathbf{e} = (i_1, i_2)$. Model $m(\mathbf{x})=\beta_0 + \beta_1 X_1 + \beta_2 X_2$ has ground-truth $\mathbf{e}^{*} = (\beta_1, \beta_2) = (0.7,0.3)$. Diverse explanations $\mathbf{e}$ map to the same quality $q$ (0, 0.5, or 1), violating local contextualization. Changing the model changes the ground-truth $\mathbf{e}^{*}$, but leaves the plots unchanged $\forall \beta_1, \beta_2$ where $\beta_1 > \beta_2 > 0$, violating model relativism. Section \ref{['subsec:gt_failures']} explains these computations.
• Figure 4: Synthetic Data and Model for AXE and PGI evaluations: 4 Normal distributions representing the data distribution, and neighborhoods $\Delta$ and $\eta$ for PGI and AXE respectively. The model is defined as $m(\mathbf{x}) = \mathbf{1}_{X_1 > 0}$, and we compare the quality of competing explanations $\mathbf{e}_a$ ($X_1$ is more important) and $\mathbf{e}_b$ ($X_2$ is more important) for datapoint $Q$.
• Figure 5: Comparing explanations using AXE and PGI: By definition $q(\mathbf{e}_a) > q(\mathbf{e}_b)$, but PGI does not clearly show this. AXE correctly determines that explanation $\mathbf{e}_a$ is better than $\mathbf{e}_b$, across hyperparameter values. (Both X axes on symlog scale).