Uncertainty Aware Neural Network from Similarity and Sensitivity

TL;DR

The paper tackles the challenge of uncertain neural network predictions by proposing a similarity- and sensitivity-aware uncertainty quantification framework. It builds uncertainty bounds via a multi-stage pipeline that learns point predictions, absolute errors, and sensitivity-weighted similarities to select relevant similar samples, supplemented by a bound-correction and a direct UB predictor NN. Empirical results across synthetic and real-world datasets show the method often outperforms Bayesian and certain direct-interval approaches, while acknowledging higher computational cost and diminished performance on very small datasets. The work provides practical tools (plots of similar samples, sample-density estimates) and public code to facilitate replication and comparison in uncertainty quantification tasks.

Abstract

Researchers have proposed several approaches for neural network (NN) based uncertainty quantification (UQ). However, most of the approaches are developed considering strong assumptions. Uncertainty quantification algorithms often perform poorly in an input domain and the reason for poor performance remains unknown. Therefore, we present a neural network training method that considers similar samples with sensitivity awareness in this paper. In the proposed NN training method for UQ, first, we train a shallow NN for the point prediction. Then, we compute the absolute differences between prediction and targets and train another NN for predicting those absolute differences or absolute errors. Domains with high average absolute errors represent a high uncertainty. In the next step, we select each sample in the training set one by one and compute both prediction and error sensitivities. Then we select similar samples with sensitivity consideration and save indexes of similar samples. The ranges of an input parameter become narrower when the output is highly sensitive to that parameter. After that, we construct initial uncertainty bounds (UB) by considering the distribution of sensitivity aware similar samples. Prediction intervals (PIs) from initial uncertainty bounds are larger and cover more samples than required. Therefore, we train bound correction NN. As following all the steps for finding UB for each sample requires a lot of computation and memory access, we train a UB computation NN. The UB computation NN takes an input sample and provides an uncertainty bound. The UB computation NN is the final product of the proposed approach. Scripts of the proposed method are available in the following GitHub repository: github.com/dipuk0506/UQ

Figures (10)

• Figure 1: Rough sketches for the visualization of the distributions of uncertainties. Sub-sketch (a) presents the distribution of targets. Sub-sketch (b) presents aleatoric, epistemic, and combined uncertainties due to insufficient data as intervals. Sub-sketches (c) to (h) present probable probability distributions of uncertainties for several different values of inputs.
• Figure 2: A parameter may not affect the prediction but may cause uncertainties. This figure shows rough sketches to visualize the influences of such parameters in uncertainty quantification. In subplot (a), Input 1 is responsible for both prediction and uncertainty. Input 2 is responsible for uncertainties. Subplots (b) and (c) present uncertainties for two different values of Input 2. Subplot (d) presents the level of quantified uncertainty when Input 2 is unknown.
• Figure 3: The reason for the development of error prediction NN on absolute error. An input parameter can be responsible for prediction or uncertainty, or both. Subplot (a) visualizes a target distribution over a range of one input. (b) visualizes targets with predictions. (c) presents prediction errors with rough values for visualization. Subplot (e) presents the mean absolute error. Uncertainty is high near 0 and 1. Uncertainty is low near 0.5. Prediction of error cannot represent the level of uncertainty where prediction of absolute error can predict that.
• Figure 4: Similar samples with the example sample and derived rough prediction interval. Similar samples and prediction intervals are plotted while (a) only similarity, (b) similarity and prediction sensitivity, (c)-(d) similarity, prediction sensitivity and error-sensitivity. In (c), prediction sensitivity and error-sensitivity have equal weights. In (d), we adjust relative weights of prediction sensitivity and error-sensitivity based on relative variances.
• Figure 5: The bound correction NN training and execution. Although similar events have very close values of highly sensitive parameters, similar events are not the exactly same event. Therefore, prediction intervals from similar events are slightly wider than actual and covers more samples than expected. Subplot (a) visualize the process of bound correction NN training. UB we want is varied from 0.01 to 0.99 with 0.01 step. Subplot (b) shows how bound correction NN helps us with modified UB values.