Deep Ensembles Secretly Perform Empirical Bayes
Gabriel Loaiza-Ganem, Valentin Villecroze, Yixin Wang
TL;DR
The paper addresses uncertainty quantification in neural networks by reconciling Bayesian neural networks (BNNs) and deep ensembles. It proves that deep ensembles perform exact Bayesian averaging through an empirically learned prior obtained via maximum marginal likelihood, i.e., empirical Bayes, and that this prior concentrates on the likelihood maximizers $\Theta^*$. Consequently, the ensemble predictive aligns with the true posterior predictive, providing a principled justification for the empirical success of ensembles and revealing that their strong priors are effectively mixtures of point masses on $\Theta^*$. This connection not only unifies two prominent UQ approaches but also suggests how priors could be structured to further enhance uncertainty estimates in practice.
Abstract
Quantifying uncertainty in neural networks is a highly relevant problem which is essential to many applications. The two predominant paradigms to tackle this task are Bayesian neural networks (BNNs) and deep ensembles. Despite some similarities between these two approaches, they are typically surmised to lack a formal connection and are thus understood as fundamentally different. BNNs are often touted as more principled due to their reliance on the Bayesian paradigm, whereas ensembles are perceived as more ad-hoc; yet, deep ensembles tend to empirically outperform BNNs, with no satisfying explanation as to why this is the case. In this work we bridge this gap by showing that deep ensembles perform exact Bayesian averaging with a posterior obtained with an implicitly learned data-dependent prior. In other words deep ensembles are Bayesian, or more specifically, they implement an empirical Bayes procedure wherein the prior is learned from the data. This perspective offers two main benefits: (i) it theoretically justifies deep ensembles and thus provides an explanation for their strong empirical performance; and (ii) inspection of the learned prior reveals it is given by a mixture of point masses -- the use of such a strong prior helps elucidate observed phenomena about ensembles. Overall, our work delivers a newfound understanding of deep ensembles which is not only of interest in it of itself, but which is also likely to generate future insights that drive empirical improvements for these models.