Implicit Generative Prior for Bayesian Neural Networks

TL;DR

The proposed NA-EB framework combines variational inference with a gradient ascent algorithm, which enables simultaneous hyperparameter selection and approximation of the posterior distribution, leading to improved computational efficiency.

Abstract

Predictive uncertainty quantification is crucial for reliable decision-making in various applied domains. Bayesian neural networks offer a powerful framework for this task. However, defining meaningful priors and ensuring computational efficiency remain significant challenges, especially for complex real-world applications. This paper addresses these challenges by proposing a novel neural adaptive empirical Bayes (NA-EB) framework. NA-EB leverages a class of implicit generative priors derived from low-dimensional distributions. This allows for efficient handling of complex data structures and effective capture of underlying relationships in real-world datasets. The proposed NA-EB framework combines variational inference with a gradient ascent algorithm. This enables simultaneous hyperparameter selection and approximation of the posterior distribution, leading to improved computational efficiency. We establish the theoretical foundation of the framework through posterior and classification consistency. We demonstrate the practical applications of our framework through extensive evaluations on a variety of tasks, including the two-spiral problem, regression, 10 UCI datasets, and image classification tasks on both MNIST and CIFAR-10 datasets. The results of our experiments highlight the superiority of our proposed framework over existing methods, such as sparse variational Bayesian and generative models, in terms of prediction accuracy and uncertainty quantification.

Figures (7)

• Figure 1: A comparison of classification results between the standard DNN method and NA-EB on four test images from the CIFAR-10 dataset.
• Figure 2: A comparison of classification results between the standard DNN method and NA-EB on four test images from the noisy MNIST with motion blur dataset.
• Figure 3: NA-EB, SGD, VBNN, and SVBNN classifying maps using 2-10-10-1 MLP, 2-20-20-1 MLP, and 2-50-50-1 MLP respectively. Black and white points denote training data for two spirals. Red and blue regions indicate the two classified classes.
• Figure 4: Difference maps of the predicted probability from four groups of weights sampled from the variational posterior of NA-EB. The yellow and dark blue areas in each map indicate the regions classified as different class by two groups of weights, respectively.
• Figure 5: Regression of two toy datasets with credible intervals using 10 sample points and 100 sample points made by NA-EB, SGD, SVBNN and VI.

Theorems & Definitions (24)

• Theorem 4.1
• Theorem 4.2
• Corollary 4.1
• Corollary 4.2
• Theorem F.1
• proof
• Theorem F.2
• proof
• Lemma 1
• proof