Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Model-Free Local Recalibration of Neural Networks

R. Torres, D. J. Nott, S. A. Sisson, T. Rodrigues, J. G. Reis, G. S. Rodrigues

TL;DR

The paper tackles uncertainty quantification for neural networks by introducing a local recalibration method that operates on layer-derived representations to correct region-specific biases in predictive distributions. It formalizes a two-stage procedure using probability integral transform (PIT) values and approximate KNN weights to produce locally calibrated predictions at any network layer, including hidden layers. Across synthetic heteroscedastic Gaussian and Gamma models, plus a diamonds dataset, the method improves MSE, interval coverage, and distributional accuracy compared with global recalibration and KNN baselines, with manageable increases in prediction time. This approach provides a scalable, practical tool for enhancing probabilistic forecasts from neural networks in real-world settings.

Abstract

Artificial neural networks (ANNs) are highly flexible predictive models. However, reliably quantifying uncertainty for their predictions is a continuing challenge. There has been much recent work on "recalibration" of predictive distributions for ANNs, so that forecast probabilities for events of interest are consistent with certain frequency evaluations of them. Uncalibrated probabilistic forecasts are of limited use for many important decision-making tasks. To address this issue, we propose a localized recalibration of ANN predictive distributions using the dimension-reduced representation of the input provided by the ANN hidden layers. Our novel method draws inspiration from recalibration techniques used in the literature on approximate Bayesian computation and likelihood-free inference methods. Most existing calibration methods for ANNs can be thought of as calibrating either on the input layer, which is difficult when the input is high-dimensional, or the output layer, which may not be sufficiently flexible. Through a simulation study, we demonstrate that our method has good performance compared to alternative approaches, and explore the benefits that can be achieved by localizing the calibration based on different layers of the network. Finally, we apply our proposed method to a diamond price prediction problem, demonstrating the potential of our approach to improve prediction and uncertainty quantification in real-world applications.

Model-Free Local Recalibration of Neural Networks

TL;DR

The paper tackles uncertainty quantification for neural networks by introducing a local recalibration method that operates on layer-derived representations to correct region-specific biases in predictive distributions. It formalizes a two-stage procedure using probability integral transform (PIT) values and approximate KNN weights to produce locally calibrated predictions at any network layer, including hidden layers. Across synthetic heteroscedastic Gaussian and Gamma models, plus a diamonds dataset, the method improves MSE, interval coverage, and distributional accuracy compared with global recalibration and KNN baselines, with manageable increases in prediction time. This approach provides a scalable, practical tool for enhancing probabilistic forecasts from neural networks in real-world settings.

Abstract

Artificial neural networks (ANNs) are highly flexible predictive models. However, reliably quantifying uncertainty for their predictions is a continuing challenge. There has been much recent work on "recalibration" of predictive distributions for ANNs, so that forecast probabilities for events of interest are consistent with certain frequency evaluations of them. Uncalibrated probabilistic forecasts are of limited use for many important decision-making tasks. To address this issue, we propose a localized recalibration of ANN predictive distributions using the dimension-reduced representation of the input provided by the ANN hidden layers. Our novel method draws inspiration from recalibration techniques used in the literature on approximate Bayesian computation and likelihood-free inference methods. Most existing calibration methods for ANNs can be thought of as calibrating either on the input layer, which is difficult when the input is high-dimensional, or the output layer, which may not be sufficiently flexible. Through a simulation study, we demonstrate that our method has good performance compared to alternative approaches, and explore the benefits that can be achieved by localizing the calibration based on different layers of the network. Finally, we apply our proposed method to a diamond price prediction problem, demonstrating the potential of our approach to improve prediction and uncertainty quantification in real-world applications.
Paper Structure (9 sections, 10 equations, 5 figures, 3 tables, 1 algorithm)

This paper contains 9 sections, 10 equations, 5 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Methods
  4. Illustrations
  5. Heteroscedastic Gaussian Model
  6. Non-Linear Gamma Model
  7. Simulation Study
  8. Diamonds Price Prediction
  9. Conclusion

Figures (5)

  • Figure 1: (a) The true relationship between the response variable and the independent variable, with the position of the highlighted observed point in relation to the predictive distribution given by the model. (b) The true mean contrasted with the model's estimated mean. (c) The predictive density of the highlighted point and its cumulative probability (PIT value). (d) The global cumulative probabilities (PIT) histogram shows the model's global bias. (e) The neighborhood of the two highlighted points. (f) The cumulative probabilities histogram of the model in the two highlighted neighborhoods shows that the model presents two distinct bias patterns depending on the region.
  • Figure 2: In Panel (a), the true and estimated means for each observation in the test set are displayed, both by the linear and the globally-recalibrated model. Panel (b) presents a similar comparison but focuses on the standard deviation. Panels (c) and (d), contrast the linear model with the locally-recalibrated model.
  • Figure 3: (a) The non-linear Rosenbrock function's surface in three dimensions, along with test data, as a function of ${\mathbf{x}}$. (b) Test data as a function of the mean. (c) WSIR predictions in the covariate space. (d) WSIR predictions before and after the recalibration. (e) 95$\%$ Confidence interval misses (light red points) throughout the covariate space for the WSIR model. (f) 95$\%$ Confidence interval misses for the WSIR model with local recalibration.
  • Figure 4: (a) and (b) show boxplots of the observed MSE for each sample and nearest neighbours samples sizes, respectively, for neural networks (NN), isotonic regression (ISO), density estimation (QNN) and our novel recalibration (RC) method. (c) and (d) present $95\%$ interval coverage for various scenarios. (e) and (f) exhibit training and prediction times, respectively, according to the sample size.
  • Figure 5: (a) Predictions variability of the base model in relation to the true data, with highlighted points $y^{(0)}_{test}$ (red) and $y^{(1)}_{test}$ (blue) for reference. (b) Estimated predictive distributions before and after recalibration for $y^{(0)}_{test}$ and $y^{(1)}_{test}$. (c) and (d) Local cumulative probabilities in the neighborhood of the highlighted points for the base and recalibrated models, respectively.