CNN-based explanation ensembling for dataset, representation and explanations evaluation

TL;DR

The paper tackles the problem that individual explanations of deep models often provide fragmented views of model behavior. It proposes a CNN-based XAI ensembling framework that consumes multiple explanation maps for each image and learns to predict segmentation masks, effectively evaluating dataset quality, representation learning, and explanation completeness by minimizing a segmentation loss $L_{\text{seg}} = \sum_i \mathrm{SoftDice}(\mathcal{M}_i, \mathrm{XAI\_Ensembler}(\mathcal{E}_i))$, yielding an ensemble explanation $\mathcal{E}^*_i$. Through experiments on ImageNet-S50 with various backbones and explanation sets, the approach demonstrates strong localisation and competitive faithfulness (as measured by Quantus metrics such as Localisation and Faithfulness), while revealing trade-offs in robustness, complexity, and randomisation depending on architecture and the number of explanations used. The method enables insights into class difficulty, potential biases, and data anonymization opportunities by highlighting how explanations capture relevant regions and suppress sensitive information, with practical implications for dataset curation and model auditing. The work also suggests future extensions to robust universal backbones and cross-modal settings, and provides a public codebase for reproducibility and broader adoption.

Abstract

Explainable Artificial Intelligence has gained significant attention due to the widespread use of complex deep learning models in high-stake domains such as medicine, finance, and autonomous cars. However, different explanations often present different aspects of the model's behavior. In this research manuscript, we explore the potential of ensembling explanations generated by deep classification models using convolutional model. Through experimentation and analysis, we aim to investigate the implications of combining explanations to uncover a more coherent and reliable patterns of the model's behavior, leading to the possibility of evaluating the representation learned by the model. With our method, we can uncover problems of under-representation of images in a certain class. Moreover, we discuss other side benefits like features' reduction by replacing the original image with its explanations resulting in the removal of some sensitive information. Through the use of carefully selected evaluation metrics from the Quantus library, we demonstrated the method's superior performance in terms of Localisation and Faithfulness, compared to individual explanations.

Figures (6)

• Figure 1: Overview of CNN-based ensembling of explanations. The diagram illustrates the process that involves an original image being processed by a black-box classification model, followed by an explainer generating explanations. These explanations, along with masks depicting objects in the original image, are then used to train XAI Ensembler, which architecture is presented in Figure \ref{['fig:architecture']}. The ensembler's training process results in an aggregated image. CNN-based ensembling of explanations with metrics we defined can be used to assess dataset, representation and explanations.
• Figure 2: Proposed three architectures of CNN-based explanation ensembling: concatenation, summation and multi-channel approach. Key architectural components are abbreviated as follows: BN (BatchNorm2D), MaxP (MaxPool2D), Conv (Conv2D), Concat (Concatenation), Sum (Summation), and SkipCon (Skip connection). The number in brackets after "ResNet" denotes the specific number of layers in the ResNet architecture.
• Figure 3: Radar plot comparing the performance of CNN-based explanation ensembling with individual non-ensembled XAI methods. The plot displays the ranking of explanations in five key evaluation metrics: Faithfulness, Robustness, Localisation, Complexity, and Randomisation. Higher values indicate better performance.
• Figure 4: A simplified illustration of introduced metrics: ensembling performance, diverseness and exhaustiveness.
• Figure 5: Averaged results of introduced metrics on the testset.