Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Post-Hoc Uncertainty Quantification in Pre-Trained Neural Networks via Activation-Level Gaussian Processes

Richard Bergna, Stefan Depeweg, Sergio Calvo Ordonez, Jonathan Plenk, Alvaro Cartea, Jose Miguel Hernandez-Lobato

TL;DR

This work tackles epistemic uncertainty in pretrained neural networks by shifting uncertainty modeling from the weight space to activation space through a post-hoc Gaussian Process, applied per neuron in the first layer (GAPA). The GP prior mean is set to the neuron's activation to preserve the network's mean predictions, while the posterior variance yields activation-level uncertainty that is deterministically propagated through the network via a delta approximation. Two variants are proposed: GAPA-Free uses empirical kernel learning for efficiency, and GAPA-Variational learns GP hyperparameters with inducing points for greater flexibility, often yielding superior uncertainty metrics compared to Laplace-based post-hoc methods. The approach preserves predictive accuracy while providing robust uncertainty estimates, with strong results on regression benchmarks and potential for scalable extensions to classification and large models.

Abstract

Uncertainty quantification in neural networks through methods such as Dropout, Bayesian neural networks and Laplace approximations is either prone to underfitting or computationally demanding, rendering these approaches impractical for large-scale datasets. In this work, we address these shortcomings by shifting the focus from uncertainty in the weight space to uncertainty at the activation level, via Gaussian processes. More specifically, we introduce the Gaussian Process Activation function (GAPA) to capture neuron-level uncertainties. Our approach operates in a post-hoc manner, preserving the original mean predictions of the pre-trained neural network and thereby avoiding the underfitting issues commonly encountered in previous methods. We propose two methods. The first, GAPA-Free, employs empirical kernel learning from the training data for the hyperparameters and is highly efficient during training. The second, GAPA-Variational, learns the hyperparameters via gradient descent on the kernels, thus affording greater flexibility. Empirical results demonstrate that GAPA-Variational outperforms the Laplace approximation on most datasets in at least one of the uncertainty quantification metrics.

Post-Hoc Uncertainty Quantification in Pre-Trained Neural Networks via Activation-Level Gaussian Processes

TL;DR

This work tackles epistemic uncertainty in pretrained neural networks by shifting uncertainty modeling from the weight space to activation space through a post-hoc Gaussian Process, applied per neuron in the first layer (GAPA). The GP prior mean is set to the neuron's activation to preserve the network's mean predictions, while the posterior variance yields activation-level uncertainty that is deterministically propagated through the network via a delta approximation. Two variants are proposed: GAPA-Free uses empirical kernel learning for efficiency, and GAPA-Variational learns GP hyperparameters with inducing points for greater flexibility, often yielding superior uncertainty metrics compared to Laplace-based post-hoc methods. The approach preserves predictive accuracy while providing robust uncertainty estimates, with strong results on regression benchmarks and potential for scalable extensions to classification and large models.

Abstract

Uncertainty quantification in neural networks through methods such as Dropout, Bayesian neural networks and Laplace approximations is either prone to underfitting or computationally demanding, rendering these approaches impractical for large-scale datasets. In this work, we address these shortcomings by shifting the focus from uncertainty in the weight space to uncertainty at the activation level, via Gaussian processes. More specifically, we introduce the Gaussian Process Activation function (GAPA) to capture neuron-level uncertainties. Our approach operates in a post-hoc manner, preserving the original mean predictions of the pre-trained neural network and thereby avoiding the underfitting issues commonly encountered in previous methods. We propose two methods. The first, GAPA-Free, employs empirical kernel learning from the training data for the hyperparameters and is highly efficient during training. The second, GAPA-Variational, learns the hyperparameters via gradient descent on the kernels, thus affording greater flexibility. Empirical results demonstrate that GAPA-Variational outperforms the Laplace approximation on most datasets in at least one of the uncertainty quantification metrics.

Paper Structure

This paper contains 16 sections, 11 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Model Proposition: GAPA + Uncertainty Propagation
  3. Pretrained Neural Network
  4. GAPA: Gausisan Process Activations
  5. Propagating the Variance through the Network
  6. Linear Transformation of Variance.
  7. Propagation Rules for Non-Linear Activations.
  8. Overall Variance Propagation.
  9. GAPA-Free: Linear Scaling of the Output Variance
  10. GAPA-Variational
  11. Results
  12. Related Work
  13. Conclusion
  14. Empirical Estimation of Inducing Inputs and RBF Kernel Hyperparameters
  15. Inducing Input Selection:
  16. ...and 1 more sections

Figures (2)

  • Figure 1: (Left) The architecture of the pre-trained backbone neural network. (Right) The GAPA module, applied post-hoc to the first layer to quantify uncertainty without modifying the original predictions. Illustration based on a toy regression problem from ortega2023variational.
  • Figure 2: Baseline activations (Top) versus GAPA activations (bottom) for neurons 1, 10, 11. GAPA preserves the mean activation while providing an uncertainty estimate.