Uncertainty Propagation in XAI: A Comparison of Analytical and Empirical Estimators

TL;DR

This paper tackles the reliability of explanations in Explainable AI by formalizing uncertainty propagation from input data and model parameters to explanations through a unified explainer function $e_\theta(x,f)$. It develops both empirical (Monte Carlo) and analytical (first-order) methods to estimate explanation variance, summarized by the Mean Uncertainty in the Explanation (MUE), and tests these approaches on MNIST and Auto MPG using five common attribution methods. The results reveal regimes where analytical and empirical estimates align, as well as scenarios where uncertainty plateaus or vanishes for small perturbations, highlighting limitations in current XAI methods’ ability to propagate uncertainty. The work underscores the importance of incorporating uncertainty into explanations, provides a general, model-agnostic framework for UXAI analysis, and discusses extensions to broader models, use cases, and non-Gaussian perturbations with practical implications for high-stakes applications.

Abstract

Understanding uncertainty in Explainable AI (XAI) is crucial for building trust and ensuring reliable decision-making in Machine Learning models. This paper introduces a unified framework for quantifying and interpreting Uncertainty in XAI by defining a general explanation function $e_θ(x, f)$ that captures the propagation of uncertainty from key sources: perturbations in input data and model parameters. By using both analytical and empirical estimates of explanation variance, we provide a systematic means of assessing the impact uncertainty on explanations. We illustrate the approach using a first-order uncertainty propagation as the analytical estimator. In a comprehensive evaluation across heterogeneous datasets, we compare analytical and empirical estimates of uncertainty propagation and evaluate their robustness. Extending previous work on inconsistencies in explanations, our experiments identify XAI methods that do not reliably capture and propagate uncertainty. Our findings underscore the importance of uncertainty-aware explanations in high-stakes applications and offer new insights into the limitations of current XAI methods. The code for the experiments can be found in our repository at https://github.com/TeodorChiaburu/UXAI

Figures (10)

• Figure 1: Empirical Monte Carlo and first order approximation approaches for estimating the propagation of uncertainty in the input $x$ (bottom) and model weights $\omega$ (top) to the explanation $e_{\theta}$.
• Figure 2: Impact of uncertainty in input variables on uncertainty of explanations. Gradient-based XAI methods exhibit a plateau in the empirical MUE on MNIST - Case 3, (b) left. Saliency and GuidedBackprop have a MUE of 0 on the regression task, when linearized - Case 2, (a). Analytical forecasts for MUE align with the empirical results for Occlusion on both datasets - Case 1, (a, b). All results are aggregated over 10 random samples from both datasets. The continuous lines show the MUE forecast computed from $\Sigma_{\text{lin}}$, while the dots mark the MUE computed from $\Sigma_{\text{MC}}$. Please beware of the different axes scales (logarithmic on the left, linear on the right). Lines not visible in the high variance regime plots are left out because of their high slope. More individual results can be seen in \ref{['fig:uncert_input_ex_tab']} and \ref{['fig:uncert_input_ex_mnist']} in the Appendix.
• Figure 3: Impact of uncertainty in model parameters on uncertainty of explanations. Empirical MUE w.r.t. perturbations in $\omega$ always follows linearization forecasts, across all tested XAI methods, all $\sigma^2$ regimes and datasets - Case 1. All results are aggregated over 10 random samples from both datasets. The continuous lines show the MUE forecast computed from $\Sigma_{\text{lin}}$, while the dots mark the MUE computed from $\Sigma_{\text{MC}}$. Please beware of the different axes scales (logarithmic on the left, linear on the right). More individual results can be seen in \ref{['fig:uncert_weights_ex_tab']} and \ref{['fig:uncert_weights_ex_mnist']} in the Appendix.
• Figure 4: Examples of input perturbations on Auto MPG. Notice the good linear fit of the analytical MUE line to the empirical MUE dots for GradientInput, Integrated Gradients and Occlusion - Case 1. The linearizations for Saliency and GuidedBackprop are 0 and not visible in the plots - Case 2. The continuous lines show the MUE forecast computed from $\Sigma_{\text{lin}}$, while the dots mark the MUE computed from $\Sigma_{\text{MC}}$. Please beware of the different axes scales (logarithmic in (a), linear in (b)).
• Figure 5: Examples of input perturbations on MNIST. Notice the good linear fit for Occlusion - Case 1, and the plateaus for the gradient-based methods in (a) - Case 3. The continuous lines show the MUE forecast computed from $\Sigma_{\text{lin}}$, while the dots mark the MUE computed from $\Sigma_{\text{MC}}$. Please beware of the different axes scales (logarithmic in (a), linear in (b) scaled by $10^{-4}$). Lines not visible in the high variance regime plots are left out because of their high slope.