Is MC Dropout Bayesian?

TL;DR

It interrogates whether MC Dropout provides a faithful Bayesian treatment for uncertainty quantification in neural networks, especially in medical imaging. It shows that MC dropout yields a multimodal, delta-Dirac posterior that assigns zero probability to the true model on simple benchmarks, challenging its Bayesian interpretation. The authors introduce a generic variational inference engine with structured normal variational families (sN-VI) and mixtures (sGMM-VI) implemented in PyTorch, designed to overcome mean-field VI limitations. Through Gaussian and RBF regression examples, they demonstrate that MC dropout can be misleading while structured VI provides more faithful posterior approximations, offering a practical no-free-lunch alternative for uncertainty quantification.

Abstract

MC Dropout is a mainstream "free lunch" method in medical imaging for approximate Bayesian computations (ABC). Its appeal is to solve out-of-the-box the daunting task of ABC and uncertainty quantification in Neural Networks (NNs); to fall within the variational inference (VI) framework; and to propose a highly multimodal, faithful predictive posterior. We question the properties of MC Dropout for approximate inference, as in fact MC Dropout changes the Bayesian model; its predictive posterior assigns $0$ probability to the true model on closed-form benchmarks; the multimodality of its predictive posterior is not a property of the true predictive posterior but a design artefact. To address the need for VI on arbitrary models, we share a generic VI engine within the pytorch framework. The code includes a carefully designed implementation of structured (diagonal plus low-rank) multivariate normal variational families, and mixtures thereof. It is intended as a go-to no-free-lunch approach, addressing shortcomings of mean-field VI with an adjustable trade-off between expressivity and computational complexity.

Figures (3)

• Figure 1: (Left) The predictive error is due to a combination of model discrepancy/bias and within-model uncertainty. E.g. parameters $\tilde{\theta}_3,\tilde{\theta}_4$ fail to be accounted for by a $2$-parameter model. The true model (green dot) is not reachable by the parametrization $\theta_1,\theta_2$. Reachable solutions lie on a $2$D manifold (green star $\equiv$ true value). The predictive posterior on $y$ (blue shades, $y$-space) reflects within-model uncertainty, i.e. uncertainty on the model parameters captured in the posterior (blue shades, $\theta$-space). (Middle) Standard Bayesian model of regression, resp. (Right) classification. Within-model uncertainty has epistemic ($\equiv$ blue node) and aleatoric ($\equiv$ orange arrow) sources.
• Figure 2: $2$D illustration of variational posterior fits, when the true posterior is unimodal (Left) or bimodal (Right). The true posterior is displayed as a blue-grey probability heatmap. Delta-Dirac masses are displayed as dots (MC dropout in red, MAP in orange). The MAP $\delta$-Dirac is at the posterior mode. MC dropout is a mixture of $2^2\!=\! 4$$\delta$-Dirac masses. The Gaussian MF-VI fit is displayed as axis-aligned isocontours.
• Figure 3: MC-dropout posterior on 1-layer RBF regression. Black line: true model. Brown dots: observations. Blue lines are not samples from the approximate posterior, but the set of $1024$ delta-Dirac functions forming the multimodal MC-dropout posterior. Evidently the location of the modes is unrelated to the model uncertainty and is instead simply an artefact of the MC-dropout approximation.