DeepUQ: Assessing the Aleatoric Uncertainties from two Deep Learning Methods

TL;DR

This work systematically compare aleatoric uncertainty measured by two UQ techniques, Deep Ensembles (DE) and Deep Evidential Regression (DER), and focuses on both zero-dimensional (0D) and two-dimensional (2D) data, to explore how the UQ methods function for different data dimensionalities.

Abstract

Assessing the quality of aleatoric uncertainty estimates from uncertainty quantification (UQ) deep learning methods is important in scientific contexts, where uncertainty is physically meaningful and important to characterize and interpret exactly. We systematically compare aleatoric uncertainty measured by two UQ techniques, Deep Ensembles (DE) and Deep Evidential Regression (DER). Our method focuses on both zero-dimensional (0D) and two-dimensional (2D) data, to explore how the UQ methods function for different data dimensionalities. We investigate uncertainty injected on the input and output variables and include a method to propagate uncertainty in the case of input uncertainty so that we can compare the predicted aleatoric uncertainty to the known values. We experiment with three levels of noise. The aleatoric uncertainty predicted across all models and experiments scales with the injected noise level. However, the predicted uncertainty is miscalibrated to $\rm{std}(σ_{\rm al})$ with the true uncertainty for half of the DE experiments and almost all of the DER experiments. The predicted uncertainty is the least accurate for both UQ methods for the 2D input uncertainty experiment and the high-noise level. While these results do not apply to more complex data, they highlight that further research on post-facto calibration for these methods would be beneficial, particularly for high-noise and high-dimensional settings.

Figures (2)

• Figure 1: Data examples for the four experimental designs: a) output uncertainty for the 0D linear regression, b) input uncertainty for 0D, c) output uncertainty for the 2D imaging data, and d) input uncertainty for the 2D data. The noise level is high for all panels: $\sigma_{y} = 0.1$. For the case of the input uncertainty panels (b and d), the uncertainty is injected on the input variable, $\sigma_{x}$, and uncertainty propagation results in a $\sigma_{y}$ value of 0.1.
• Figure 2: Distribution of predicted $\sigma_{\rm al}$ values. The circular point for each distribution is the sample mean and the black error bars show the $\rm {std}(\sigma_{\rm al})$ confidence range, the standard deviation of the $\sigma_{\rm al}$ distribution. The vertical dashed lines demonstrate the true output uncertainty values, $\sigma_y$, which vary by noise level. The light pink, medium pink, and purple distributions correspond to the low-, medium-, and high-noise models.