UbiQVision: Quantifying Uncertainty in XAI for Image Recognition

TL;DR

UbiQVision addresses the critical need to quantify uncertainty in XAI for medical imaging by unifying Bayesian model weighting, SHAP-based attribution, and Dempster-Shafer fusion. The framework converts pixel-level SHAP scores into evidential masses, weighted by validated model reliability, and fused via DST to produce Belief, Plausibility, and Uncertainty maps. It demonstrates improved localization of pathology and explicit ignorance signals across malaria, Alzheimer's, and diabetic retinopathy datasets, offering robust, interpretable outputs for clinical decision-making. This approach enhances trust and safety in AI-driven diagnostics and aligns with emerging regulatory demands for reliability and transparency in medical AI.

Abstract

Recent advances in deep learning have led to its widespread adoption across diverse domains, including medical imaging. This progress is driven by increasingly sophisticated model architectures, such as ResNets, Vision Transformers, and Hybrid Convolutional Neural Networks, that offer enhanced performance at the cost of greater complexity. This complexity often compromises model explainability and interpretability. SHAP has emerged as a prominent method for providing interpretable visualizations that aid domain experts in understanding model predictions. However, SHAP explanations can be unstable and unreliable in the presence of epistemic and aleatoric uncertainty. In this study, we address this challenge by using Dirichlet posterior sampling and Dempster-Shafer theory to quantify the uncertainty that arises from these unstable explanations in medical imaging applications. The framework uses a belief, plausible, and fusion map approach alongside statistical quantitative analysis to produce quantification of uncertainty in SHAP. Furthermore, we evaluated our framework on three medical imaging datasets with varying class distributions, image qualities, and modality types which introduces noise due to varying image resolutions and modality-specific aspect covering the examples from pathology, ophthalmology, and radiology, introducing significant epistemic uncertainty.

Figures (10)

• Figure 1: On the left, the plot demonstrates the impact of the temperature parameter (T) on the model weights for Models A, B, and C, as evaluated on the test dataset using a specific metric on a logarithmic scale. A low T value means that a model with even a slightly higher F1 score receives the highest weight, while a high T value means that the models will be allotted equal weight, irrespective of their performance. On the right, the plot shows the impact of the sensitivity parameter $\lambda$ on the generated belief mass function. This parameter controls the signal-to-noise ratio in the uncertainty maps. When $\lambda$ is low, the resultant maps will assume that everything is noise. With a high $\lambda$, the very low background noise would be counted as maximum belief, resulting in a binary mask with no useful uncertainty maps.
• Figure 2: The visualization shows the SHAP values for the two classes of the malaria dataset from the predictions of three models: a custom CNN model, a ResNet model, and a ViT model, with weights of 0.37, 0.32, and 0.31, respectively. The "uninfected" class receives positive attribution ($\phi > 0$) from the models for the erythrocyte’s interior regions with low-frequency spatial variation. Specifically, the smooth, homogeneous cytoplasm receives positive attribution. Conversely, the models assign high negative importance ($\phi < 0$) to high-contrast, high-frequency anomalies, such as ring-like chromatin structures.
• Figure 3: This figure shows the fusion of SHAP explanations from the weighted model ensemble (Custom CNN, ResNet, and ViT) for a malaria-infected and uninfected sample. The figure (a) is of the parasitized sample. The belief mass (support) map shows a concentrated area of high belief (dark green), which precisely localizes the parasite. This indicates that the models found strong, consistent evidence at this location. This confidence is reflected in the uncertainty (ignorance) map, where the central purple region represents a significant reduction in uncertainty (approaching 0.0). According to Dempster-Shafer theory, this reduction in uncertainty indicates that the models are highly confident and in agreement about the feature of interest. In contrast, the surrounding cytoplasm (yellow) has high epistemic uncertainty due to a lack of distinguishing features. The figure (b) is of an uninfected sample. This figure illustrates the fusion results for a healthy, uninfected cell, which contrasts sharply with the parasitized sample. The belief mass map appears diffuse and scattered, with no strong, centralized support features. This correctly reflects the absence of a foreign body (parasite). Consequently, the uncertainty (ignorance) map demonstrates high uncertainty (bright yellow) across the entire cell structure. This uniform distribution of ignorance indicates that, although the ensemble likely predicts the class correctly, the decision relies on the general absence of features rather than the detection of a specific object. This results in higher overall quantified uncertainty compared to the infected sample.
• Figure 4: Figure (a) shows the distribution of evidence and model confidence for the parasitized sample. It presents the statistical analysis of the fusion process for the infected erythrocyte. The left panel shows the kernel density estimation (KDE) of pixel-wise mass values. The belief mass (green) is heavily skewed toward zero, but it has a noticeable tail that extends into higher values. This statistically confirms the presence of a specific, localized region of strong support (the parasite) amidst a background of low belief. The uncertainty (purple) shows wider variance, reflecting confident detection in the center and high ignorance in the cytoplasm. The right panel shows the Bayesian weights assigned to the ensemble via a Dirichlet distribution based on validation F1 scores. In this case, the custom CNN received the highest posterior weight (w = 0.366), followed by ResNet (w = 0.320) and ViT (w = 0.315). This indicates that the simpler architecture offered slightly higher robustness for this specific split. Figure (b) shows the statistical profile of uninfected cell fusion and clearly contrasts with the parasitized analysis. In the KDE plot (left), the belief mass (green) is compressed near 0.0 with no significant tail. This quantitatively verifies that the models found almost no pixel-wise evidence to support the parasitized class. Consequently, the uncertainty (purple) and plausibility (blue) distributions are tightly clustered near 1.0, signifying high global ignorance. This confirms that the system's decision was driven by a lack of positive evidence rather than the detection of contradictory features. The Bayesian model confidence (right) remains consistent with the previous figure and illustrates the fixed weighting scheme applied across the dataset during the inference phase.
• Figure 5: This visualization shows the SHAP attribution maps ($\phi$) for each dementia stage. It details the additive feature attribution scores for the ensemble members, where the color intensity corresponds to the impact on the model's log-odds output. Red pixels denote positive SHAP values (phi > 0), indicating morphological regions, such as enlarged ventricles or cortical atrophy, that drive classification toward a specific dementia stage. Conversely, blue pixels indicate negative SHAP values ($\phi < 0$), representing features that lower the probability of the target class. The spatial distribution of the SHAP values varies significantly by architecture. For example, the ResNet model frequently exhibits scattered, high-frequency pixel importance, while the Custom CNN and ViT models display contiguous regions of semantic relevance. This demonstrates how distinct internal representations lead to different pixel-wise evidence-gathering strategies.

Theorems & Definitions (15)

• Definition 1: Prior Distribution
• Definition 2: Likelihood
• Theorem 1: Posterior Derivation
• proof : Proof of Convergence
• Definition 3: Additive Feature Attribution
• Remark 1
• Definition 4: BPA Mapping Function
• Proposition 1: Validity of Mass Function
• proof
• Definition 5: Dempster's Rule of Combination