Model and Feature Diversity for Bayesian Neural Networks in Mutual Learning

TL;DR

The paper tackles the gap where Bayesian neural networks struggle to match deterministic models by introducing a deep mutual learning framework that promotes diversity in both model-parameter distributions and intermediate feature representations among peer BNNs. It formalizes parameter-space diversity with distance measures between Gaussian posteriors and encourages broader feature coverage by using fused features and distribution-based divergence. The approach yields consistent improvements in accuracy, negative log-likelihood, and calibration on CIFAR-10/100 and ImageNet, and it also enhances uncertainty estimation as evaluated by BALD. This work demonstrates that incorporating parameter-space diversity into mutual learning substantially strengthens BNN performance and uncertainty reliability, with potential for practical deployment in uncertainty-aware deep learning tasks.

Abstract

Bayesian Neural Networks (BNNs) offer probability distributions for model parameters, enabling uncertainty quantification in predictions. However, they often underperform compared to deterministic neural networks. Utilizing mutual learning can effectively enhance the performance of peer BNNs. In this paper, we propose a novel approach to improve BNNs performance through deep mutual learning. The proposed approaches aim to increase diversity in both network parameter distributions and feature distributions, promoting peer networks to acquire distinct features that capture different characteristics of the input, which enhances the effectiveness of mutual learning. Experimental results demonstrate significant improvements in the classification accuracy, negative log-likelihood, and expected calibration error when compared to traditional mutual learning for BNNs.