Identity Curvature Laplace Approximation for Improved Out-of-Distribution Detection

TL;DR

The Identity Curvature Laplace Approximation (ICLA), a novel method that challenges the conventional posterior covariance formulation by using identity curvature and optimizing prior precision, is introduced.

Abstract

Uncertainty estimation is crucial in safety-critical applications, where robust out-of-distribution (OOD) detection is essential. Traditional Bayesian methods, though effective, are often hindered by high computational demands. As an alternative, Laplace approximation offers a more practical and efficient approach to uncertainty estimation. In this paper, we introduce the Identity Curvature Laplace Approximation (ICLA), a novel method that challenges the conventional posterior covariance formulation by using identity curvature and optimizing prior precision. This innovative design significantly enhances OOD detection performance on well-known datasets such as CIFAR-10, CIFAR-100, and ImageNet, while maintaining calibration scores. We attribute this improvement to the alignment issues between typical feature embeddings and curvature as measured by the Fisher information matrix. Our findings are further supported by demonstrating that incorporating Fisher penalty or sharpness-aware minimization techniques can greatly enhance the uncertainty estimation capabilities of standard Laplace approximation.

Figures (11)

• Figure 1: Identity Curvature Laplace Approximation (ICLA), without Hessian computation, produces smoother uncertainty landscapes than standard last-layer Laplace approximation (LLLA), and leads to improved OOD detection performance in real-world scenarios.
• Figure 2: Uncertainty estimation on a toy binary classification dataset with test outliers for MAP network (left), Standard LLLA (middle), ICLA (right). For each setup, the average predictive uncertainty for outlier samples is displayed in the top-left corner. It can be seen that using identity curvature and prior precision as the source of covariance leads to a wider uncertainty surface while preserving the posterior landscape. Subsequently, it achieves better entropy estimates for outlier data points. See Section \ref{['sec:toy']}.
• Figure 3: Uncertainty estimation on a toy sinusoidal regression dataset for MAP network (left), Standard LLLA (middle), ICLA (right). The observed property of Laplace approximation and identity curvature generalizes to regression tasks and does not negatively impact uncertainty estimates. See Section \ref{['sec:toy']}.
• Figure 4: Calibration scores. Expected calibration error (ECE), negative log-likelihood (NLL), and Brier score for CIFAR-10 (left), CIFAR-100 (middle) and ImageNet-200 (right) for different Laplace approximations. Identity curvature does not detrimentally affect model calibration. See Section \ref{['sec:acc_and_calib']} for details.
• Figure 5: Epoch-wise comparison between standard LLLA and ICLA for OOD detection. Based on this experiment, we can conclude that the observed phenomenon is unrelated to the model training stage. See Section \ref{['sec:training_stage']} for details.