Quantification of Uncertainties in Probabilistic Deep Neural Network by Implementing Boosting of Variational Inference

TL;DR

The paper addresses the challenge of uncertainty quantification in deep neural networks by moving beyond point-weight representations to probabilistic weights. It introduces Boosted Bayesian Neural Networks (BBNN) that integrate Boosting Variational Inference (BVI) with reparameterization-based training (including Bayes By Backpropagation) to form a flexible mixture posterior $q(W)$ that better approximates the true posterior $p(W|D)$. Empirical results on medical datasets show that BBNN can improve predictive accuracy in some domains and consistently provide sharper, better-calibrated uncertainty (lower NLL and ECE), albeit at higher computational cost and with some dataset-specific limitations. This approach advances probabilistic deep learning by enabling multimodal posterior approximations, improved uncertainty estimation, and robustness in data-scarce, high-stakes settings. The work has practical implications for domains like medical diagnostics where reliable uncertainty is crucial for decision-making and risk assessment.

Abstract

Modern neural network architectures have achieved remarkable accuracies but remain highly dependent on their training data, often lacking interpretability in their learned mappings. While effective on large datasets, they tend to overfit on smaller ones. Probabilistic neural networks, such as those utilizing variational inference, address this limitation by incorporating uncertainty estimation through weight distributions rather than point estimates. However, standard variational inference often relies on a single-density approximation, which can lead to poor posterior estimates and hinder model performance. We propose Boosted Bayesian Neural Networks (BBNN), a novel approach that enhances neural network weight distribution approximations using Boosting Variational Inference (BVI). By iteratively constructing a mixture of densities, BVI expands the approximating family, enabling a more expressive posterior that leads to improved generalization and uncertainty estimation. While this approach increases computational complexity, it significantly enhances accuracy an essential tradeoff, particularly in high-stakes applications such as medical diagnostics, where false negatives can have severe consequences. Our experimental results demonstrate that BBNN achieves ~5% higher accuracy compared to conventional neural networks while providing superior uncertainty quantification. This improvement highlights the effectiveness of leveraging a mixture-based variational family to better approximate the posterior distribution, ultimately advancing probabilistic deep learning.

Figures (5)

• Figure 1: Classical Neural Network with Point Weights and Neural Network with Distribution of Weights
• Figure 2: Optimization of the true 1-Dimensional posterior after adding components in Boosting Variational Inference.
• Figure 3: Predictive distributions on the Diabetes dataset. (a) displays the output from standard Variational Inference (VI) with relatively broad confidence intervals, while (b) shows the refined predictive distribution from Boosted VI Neural Networks (BBNN).
• Figure 4: Visualization of the refined posterior distributions achieved by the boosting mechanism in BBNN.
• Figure 5: Visualization of the refined posterior approximation achieved by Boosted VI. The figure illustrates how iterative boosting steps lead to a more expressive approximation that closely matches the true posterior distribution.