Explaining Predictive Uncertainty by Exposing Second-Order Effects

TL;DR

This investigation reveals that predictive uncertainty is dominated by second-order effects, involving single features or product interactions between them, and contributes a new method for explaining predictive uncertainty based on these second-order effects.

Abstract

Explainable AI has brought transparency into complex ML blackboxes, enabling, in particular, to identify which features these models use for their predictions. So far, the question of explaining predictive uncertainty, i.e. why a model 'doubts', has been scarcely studied. Our investigation reveals that predictive uncertainty is dominated by second-order effects, involving single features or product interactions between them. We contribute a new method for explaining predictive uncertainty based on these second-order effects. Computationally, our method reduces to a simple covariance computation over a collection of first-order explanations. Our method is generally applicable, allowing for turning common attribution techniques (LRP, Gradient x Input, etc.) into powerful second-order uncertainty explainers, which we call CovLRP, CovGI, etc. The accuracy of the explanations our method produces is demonstrated through systematic quantitative evaluations, and the overall usefulness of our method is demonstrated via two practical showcases.

Figures (5)

• Figure 1: Illustration of an ensemble model (top left), its prediction and predictive uncertainty (top right) and an illustrative cartoon example of our proposed predictive uncertainty explanation in terms of features and feature interactions (bottom). Red patches and connecting lines highlight features and feature interactions adding to predictive uncertainty; blue connecting lines highlight feature interactions decreasing predictive uncertainty.
• Figure 2: Left: Classical explanation workflow, commonly used for attributing the output of a neural network model to the individual input features (elements of $x$). Right: Proposed explanation method for explaining predictive uncertainty. Predictive uncertainty (estimated by the variance over an ensemble's predictions) is attributed to elements of $xx^\top$ (a second-order explanation) by computing a covariance over explanations associated to each member of the ensemble.
• Figure 3: Comparison of the uncertainty attribution of the original and the fine-tuned ensemble for two different underrepresented features. In the upper row, the original ensemble was trained without 'eyeglass' images; in the lower row, the original ensemble was trained without 'hat' images. Left: T-SNE visualization of the original ensemble's hidden activations for the test data points, with the actual class labels colored in green-blue and the experimental set in red. Middle: Share of uncertainty attributed to different visual features for the original and the fine-tuned ensemble, highlighting the primary role of 'eyeglass' and 'hat' features in reducing uncertainty. Right: Heatmap explanations of the original and the fine-tuned ensemble for three images from the experimental data, illustrating on a pixel-wise basis the reduction of 'hat' and 'eyeglass' features as contributors to uncertainty.
• Figure 4: Predictive uncertainty of day-ahead price prediction and uncertainty relevance analysis of three high-uncertainty days. The upper plot depicts the trained deep ensemble's hourly predictive uncertainty over the course of an entire month. The lower plot depicts for three consecutive days the predictive uncertainty and the 24 last values of the input channels $x_1,x_2,x_3$. Additionally, the CovLRP attribution of uncertainty onto these three channels is depicted in shades of red for diagonal terms, and as two-sided arrows for off-diagonal terms capturing the highest interactions. Solid connecting lines denote strong interactions and dashed connecting lines denote weaker interactions.
• Figure 5: Contribution of residual demand to uncertainty in February 2020. The upper plot depicts the trained deep ensemble's predictive uncertainty. The lower plot depicts residual demand, calculated as the difference between demand ($x_3$) and renewable production ($x_2$), and the shades of red show the sum of contribution to uncertainty associated to these two features. The figure shows a negative correlation between residual demand and uncertainty and the explanation's focus on low residual demand periods.

Theorems & Definitions (4)

• Proposition 1: Conservation
• Proposition 2: Preservation of Irrelevance
• Proposition 3
• Proposition 4