Rethinking Explanation Evaluation under the Retraining Scheme

TL;DR

This paper tackles the unreliable evaluation of feature attributions for explainable AI by analyzing distortion in the ROAR retraining scheme, pinpointing residual information and a Sign issue as key culprits. It reframes the evaluation with Keep and Fine-tune (kaft) and head-only variants (kaft-c, |raft-c|), achieving substantial efficiency while preserving alignment with explanation quality. Empirical results across small and large scales show IG and SIG often outperform others under the new schemes, though failures on SwinT reveal deeper issues like gradient saturation and interaction effects that warrant further study. The proposed framework offers a practical, theory-grounded alternative to inference-based evaluation and facilitates more reliable explainer benchmarking and selection.

Abstract

Feature attribution has gained prominence as a tool for explaining model decisions, yet evaluating explanation quality remains challenging due to the absence of ground-truth explanations. To circumvent this, explanation-guided input manipulation has emerged as an indirect evaluation strategy, measuring explanation effectiveness through the impact of input modifications on model outcomes during inference. Despite the widespread use, a major concern with inference-based schemes is the distribution shift caused by such manipulations, which undermines the reliability of their assessments. The retraining-based scheme ROAR overcomes this issue by adapting the model to the altered data distribution. However, its evaluation results often contradict the theoretical foundations of widely accepted explainers. This work investigates this misalignment between empirical observations and theoretical expectations. In particular, we identify the sign issue as a key factor responsible for residual information that ultimately distorts retraining-based evaluation. Based on the analysis, we show that a straightforward reframing of the evaluation process can effectively resolve the identified issue. Building on the existing framework, we further propose novel variants that jointly structure a comprehensive perspective on explanation evaluation. These variants largely improve evaluation efficiency over the standard retraining protocol, thereby enhancing practical applicability for explainer selection and benchmarking. Following our proposed schemes, empirical results across various data scales provide deeper insights into the performance of carefully selected explainers, revealing open challenges and future directions in explainability research.

Figures (11)

• Figure 1: Example of the Sign issue. The first two columns show the original input and the corresponding feature attributions derived by IG. The subsequent columns present the manipulated inputs after removing $90\%$ of features following three evaluation schemes: roar, kear, and $\vert$roar$\vert$. For roar, the target-overlapping negative features leak information about the target object, leading to evaluation distortion. Despite the visual similarity of the manipulated inputs, retraining following kear achieves over $20\%$ higher accuracy, implying that the explainer correctly identifies contributing features with an appropriate class association. As a control group, removing all relevant features (by $\vert$roar$\vert$) achieves the expectation described in Equation \ref{['eq:roar_exp']} and yields the lowest accuracy among the three, which is $6\%$ lower than random feature removal.
• Figure 2: Example of a weak positive contributor. The feature $x_i$ is a weak contributor to class $1$: it positively influences $o_1$ but yields a negative contribution to $f_1$.
• Figure 3: Visualization of model updates under different evaluation schemes. The inference scheme is sensitive to distribution shifts as it does not update the tested model. kaft and its variants overcome this limitation while mitigating distortions inherent in other retraining schemes. Particularly, kaft-c evaluates explanation quality by assessing the utility of preserved features through restricted updates, striking a balance between resolving distribution shifts and maintaining focus on model behavior.
• Figure 4: Explainer performance reflected by different schemes with $90\%$ of features removed. Evaluation results of each scheme are grouped by test case.
• Figure 5: Explanation quality in small-scale settings tested by kaft-c and $\vert$raft-c$\vert$. The embedded tables present $\Delta\textrm{Acc.}$, which quantifies explainer performance. $\Delta\textrm{Acc.}$ represents the difference in prediction power, computed as the area between the degradation curves produced by kaft-c and $\vert$raft-c$\vert$.

Theorems & Definitions (7)

• Definition 1: Secondary Evidence
• Theorem 1: Increasing Utility of Secondary Features
• Definition 2: Weak Positive Contributor
• Theorem 2: Negative Attribution
• Theorem 2: Negative Attribution
• proof : Proof of Theorem \ref{['thm:swap']}
• proof : Proof of Theorem \ref{['thm:wpc']}