Evidential Deep Learning: Enhancing Predictive Uncertainty Estimation for Earth System Science Applications

TL;DR

This study compares the uncertainty derived from evidential neural networks to that obtained from ensembles and shows evidential deep learning models attaining predictive accuracy rivaling standard methods, while robustly quantifying both sources of uncertainty.

Abstract

Robust quantification of predictive uncertainty is critical for understanding factors that drive weather and climate outcomes. Ensembles provide predictive uncertainty estimates and can be decomposed physically, but both physics and machine learning ensembles are computationally expensive. Parametric deep learning can estimate uncertainty with one model by predicting the parameters of a probability distribution but do not account for epistemic uncertainty.. Evidential deep learning, a technique that extends parametric deep learning to higher-order distributions, can account for both aleatoric and epistemic uncertainty with one model. This study compares the uncertainty derived from evidential neural networks to those obtained from ensembles. Through applications of classification of winter precipitation type and regression of surface layer fluxes, we show evidential deep learning models attaining predictive accuracy rivaling standard methods, while robustly quantifying both sources of uncertainty. We evaluate the uncertainty in terms of how well the predictions are calibrated and how well the uncertainty correlates with prediction error. Analyses of uncertainty in the context of the inputs reveal sensitivities to underlying meteorological processes, facilitating interpretation of the models. The conceptual simplicity, interpretability, and computational efficiency of evidential neural networks make them highly extensible, offering a promising approach for reliable and practical uncertainty quantification in Earth system science modeling. In order to encourage broader adoption of evidential deep learning in Earth System Science, we have developed a new Python package, MILES-GUESS (https://github.com/ai2es/miles-guess), that enables users to train and evaluate both evidential and ensemble deep learning.

Figures (23)

• Figure 1: (a-b) Example of a precipitation type sounding with temperature, dewpoint, and wind profiles along with (c-d) different predictive uncertainty representations.
• Figure 2: (a) Deterministic and evidential MLP architectures for predicting class probabilities in the precipitation-type categorical dataset. (b) Architectures for predicting parameters in the surface-layer regression dataset, including Gaussian ($\mu$, $\sigma^2$) and Normal-Inverse Gaussian ($\gamma$, $\nu$, $\alpha$, $\beta$) distributions. In both architecture diagrams, the neural network is represented by $\text{f}_\textbf{w}$ where $\textbf{w}$ are trainable parameters.
• Figure 3: Attributes diagrams for the (a) the evidential model and (b) a deterministic neural network model. The columns show the result for each precipitation-type. In each sub-panel, the diagonal, horizontal, and vertical dashed lines indicate the 1-to-1 line, no-resolution line, and climatology line. The blue-shaded area indicates skill relative to climatology. Red-shaded rectangles illustrate the reliability of each model in predicting each class. The legend in each panel displays the Brier Skill Score, along with the reliability (Rel) and resolution (Res) components of the Brier score scaled by the uncertainty component. Reliability describes the deviation of the predicted probability from the observed relative frequency (lower is better) and Resolution describes the average difference in the predicted probabilities from climatology, related to sharpness where higher values are better.
• Figure 4: 2D histograms illustrate the relationship between the top-1 aleatoric and top-1 epistemic uncertainties for (a) the MC-ensemble deterministic model and (b) the evidential model. In (c) and (d), the total summed quantities for these uncertainties are compared.
• Figure 5: 2D histograms compare aleatoric and epistemic uncertainty estimates computed using the LoTV and evidential $u$ from DST. Figures (a) and (b) plot the top-1 predicted aleatoric and epistemic uncertainties, respectively, against $u$. Figures (c) and (d) show the total aleatoric and total epistemic uncertainties, summed over all classes, versus $u$. The dashed line indicates a 1-to-1 relationship in each panel.