Solving the enigma: Enhancing faithfulness and comprehensibility in explanations of deep networks

TL;DR

The paper tackles the problem of opaque deep networks and inconsistent explanations across XAI methods. It introduces an explanation optimizer—a non-linear encoder–decoder that fuses outputs from $K$ baseline XAI methods and optimizes a loss combining $M_{faith}$, $M_{compx}$, and a similarity term to a weighted baseline, achieving high-fidelity, low-complexity explanations. Validations on 3D TOP-OSLO brain imaging and 2D CIFAR-10 show substantial improvements in faithfulness (up to $155\%$ over the best method) while reducing complexity, with high-resolution explanations produced via up-sampling. The framework generalizes to multi-dimensional inputs and holds promise for reducing inter-method variability, thereby enhancing trust and adoption of AI predictions in critical domains.

Abstract

The accelerated progress of artificial intelligence (AI) has popularized deep learning models across various domains, yet their inherent opacity poses challenges, particularly in critical fields like healthcare, medicine, and the geosciences. Explainable AI (XAI) has emerged to shed light on these 'black box' models, aiding in deciphering their decision-making processes. However, different XAI methods often produce significantly different explanations, leading to high inter-method variability that increases uncertainty and undermines trust in deep networks' predictions. In this study, we address this challenge by introducing a novel framework designed to enhance the explainability of deep networks through a dual focus on maximizing both accuracy and comprehensibility in the explanations. Our framework integrates outputs from multiple established XAI methods and leverages a non-linear neural network model, termed the 'explanation optimizer,' to construct a unified, optimal explanation. The optimizer evaluates explanations using two key metrics: faithfulness (accuracy in reflecting the network's decisions) and complexity (comprehensibility). By balancing these, it provides accurate and accessible explanations, addressing a key XAI limitation. Experiments on multi-class and binary classification in 2D object and 3D neuroscience imaging confirm its efficacy. Our optimizer achieved faithfulness scores 155% and 63% higher than the best XAI methods in 3D and 2D tasks, respectively, while also reducing complexity for better understanding. These results demonstrate that optimal explanations based on specific quality criteria are achievable, offering a solution to the issue of inter-method variability in the current XAI literature and supporting more trustworthy deep network predictions

Figures (7)

• Figure 1: Deriving optimal local explanations for deep classifiers in anatomical neuroscience. The figure depicts the traditional local explanation process (implementing multiple XAI methods) for a three-dimensional binary classification example regarding the existence or absence of PCS (ParaCingulate Sulcus). This is one of the two case studies we consider in this paper. Note that the high variability in the local explanations from different XAI methods increases uncertainty and lowers trust in explainability among experts. Therefore, an explanation optimizer is proposed to derive a unique and optimal explanation for the pre-trained Deep Classifier in the computer vision task of interest.
• Figure 2: The proposed framework for optimizing explanations in the 3D task. The key components of the framework include the utilization of $K$ baseline XAI methods (K methods), a 'Weighted Average' explanation, a non-linear network ('Non-linear structure'), and an 'Up-Sampling' unit. The framework incorporates a range of explanations obtained from established XAI methods and computes an adjusted baseline explanation ('Weighted Average'). The local explanations and the 'Weighted Average' are concatenated and fed into a suitably designed non-linear structure (Swin-UNETR). The reconstructed optimal explanation feeds an upsampling deconvolution layer to increase the resolution (high-resolution (HR) local explanation). The cost function evaluates the similarity between the weighted average and the low-resolution prediction, which is added to the faithfulness and complexity scores of the explanation. The 3D application uses images from the Top-Oslo dataset mo1.
• Figure 3: Training results for the 3D classification task: Detecting the existence/absence of PCS. (a) The curves and light blue shading in the panels highlight the median and min/max loss values, respectively, obtained during training across various learning rates. The utilized network is Resnet-18, trained and validated in the Top-Oslo cohort. The loss based on training (validation) data is shown on the right (left) panel. (b) Same as in (a), but the classification accuracy is presented.
• Figure 4: Explanation optimization for the 3D application: Training Results. The curves and light blue shading in the panels highlight the median and the min/max loss values, respectively, obtained during training across various learning rates. The two panels present the validation loss of the explanation optimizer for the binary classification task for class 0 (absence of PCS; left panel) and class 1 (existence of PCS; right panel) in the 3D brain application.
• Figure 5: Box plot results comparing state-of-the-art XAI methods with the proposed explanation optimizer for the 3D application in the testing cohort. (a) The faithfulness score of different state-of-the-art XAI methods (green color variation) alongside the proposed non-linear explanation optimizer (red color) for the 3D application. (b) Same as in (a), but for the complexity score.