When Can You Trust Your Explanations? A Robustness Analysis on Feature Importances
Ilaria Vascotto, Alex Rodriguez, Alessandro Bonaita, Luca Bortolussi
TL;DR
This paper tackles the challenge of trustworthy explanations by formalizing robustness of feature importances in neural networks under non-adversarial, on-manifold perturbations. It introduces a robustness estimator based on Spearman rank correlation between explanations before and after perturbations, and evaluates it within a pipeline that includes an ensemble of three local, post-hoc explainers (DeepLIFT, Integrated Gradients, and Layerwise Relevance Propagation) applied to tabular data. A novel neighbourhood generation strategy along the data manifold, plus an ensemble aggregation and a kNN-based reliability check, enable reliable detection of robust, uncertain, and non-robust explanations. The approach is validated on eight public tabular datasets, demonstrating that robust explanations tend to cluster in robust regions of the input space and that ensemble explanations generally provide more stable robustness signals than any single method. Overall, the framework offers practitioners a practical, ground-truth-free means to evaluate and trust explanations in real-world applications.
Abstract
Recent legislative regulations have underlined the need for accountable and transparent artificial intelligence systems and have contributed to a growing interest in the Explainable Artificial Intelligence (XAI) field. Nonetheless, the lack of standardized criteria to validate explanation methodologies remains a major obstacle to developing trustworthy systems. We address a crucial yet often overlooked aspect of XAI, the robustness of explanations, which plays a central role in ensuring trust in both the system and the provided explanation. To this end, we propose a novel approach to analyse the robustness of neural network explanations to non-adversarial perturbations, leveraging the manifold hypothesis to produce new perturbed datapoints that resemble the observed data distribution. We additionally present an ensemble method to aggregate various explanations, showing how merging explanations can be beneficial for both understanding the model's decision and evaluating the robustness. The aim of our work is to provide practitioners with a framework for evaluating the trustworthiness of model explanations. Experimental results on feature importances derived from neural networks applied to tabular datasets highlight the importance of robust explanations in practical applications.