Improving the Linearized Laplace Approximation via Quadratic Approximations

TL;DR

The paper tackles overconfident out-of-distribution predictions by improving the Linearized Laplace Approximation (LLA) through Quadratic Laplace Approximation (QLA). QLA introduces a second-order (quadratic) expansion around the MAP $\theta^*$ and uses a rank-one Hessian approximation $\hat{\mathbf{z}}\hat{\mathbf{z}}^\top$ obtained via power iterations on Hessian-vector products, avoiding the full Hessian while yielding a posterior precision closer to the full Laplace. Predictions retain the stable linearized form to mitigate mass allocations in unsupported regions, resulting in a modest but consistent improvement in uncertainty metrics (NLL, CRPS) across five regression datasets with negligible extra cost. This enhances calibrated uncertainty in Bayesian neural network predictions without compromising predictive performance, and lays groundwork for extensions to classification and scalable large-scale models.

Abstract

Deep neural networks (DNNs) often produce overconfident out-of-distribution predictions, motivating Bayesian uncertainty quantification. The Linearized Laplace Approximation (LLA) achieves this by linearizing the DNN and applying Laplace inference to the resulting model. Importantly, the linear model is also used for prediction. We argue this linearization in the posterior may degrade fidelity to the true Laplace approximation. To alleviate this problem, without increasing significantly the computational cost, we propose the Quadratic Laplace Approximation (QLA). QLA approximates each second order factor in the approximate Laplace log-posterior using a rank-one factor obtained via efficient power iterations. QLA is expected to yield a posterior precision closer to that of the full Laplace without forming the full Hessian, which is typically intractable. For prediction, QLA also uses the linearized model. Empirically, QLA yields modest yet consistent uncertainty estimation improvements over LLA on five regression datasets.