Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is BatchEnsemble a Single Model? On Calibration and Diversity of Efficient Ensembles

Anton Zamyatin, Patrick Indri, Sagar Malhotra, Thomas Gärtner

TL;DR

The paper addresses whether BatchEnsemble can provide true ensemble-level uncertainty with substantially fewer parameters by using rank-1 perturbations of a shared network. The authors theoretically show that BatchEnsemble’s perturbations access only a measure-zero subset of the full Deep Ensemble weight space, and empirically demonstrate that BatchEnsemble behaves like a single model with limited epistemic uncertainty across CIFAR-10/10C/SVHN and MNIST. Deep Ensembles consistently outperform BatchEnsemble in accuracy, calibration, and OOD detection, while BatchEnsemble’s diversity metrics remain near those of a single model. The work highlights the importance of reporting uncertainty diagnostics such as predictive entropy and Jensen–Shannon divergence to assess true predictive diversity and questions the practical utility of BatchEnsemble as a drop-in replacement for true ensembles in resource-constrained settings.

Abstract

In resource-constrained and low-latency settings, uncertainty estimates must be efficiently obtained. Deep Ensembles provide robust epistemic uncertainty (EU) but require training multiple full-size models. BatchEnsemble aims to deliver ensemble-like EU at far lower parameter and memory cost by applying learned rank-1 perturbations to a shared base network. We show that BatchEnsemble not only underperforms Deep Ensembles but closely tracks a single model baseline in terms of accuracy, calibration and out-of-distribution (OOD) detection on CIFAR10/10C/SVHN. A controlled study on MNIST finds members are near-identical in function and parameter space, indicating limited capacity to realize distinct predictive modes. Thus, BatchEnsemble behaves more like a single model than a true ensemble.

Is BatchEnsemble a Single Model? On Calibration and Diversity of Efficient Ensembles

TL;DR

The paper addresses whether BatchEnsemble can provide true ensemble-level uncertainty with substantially fewer parameters by using rank-1 perturbations of a shared network. The authors theoretically show that BatchEnsemble’s perturbations access only a measure-zero subset of the full Deep Ensemble weight space, and empirically demonstrate that BatchEnsemble behaves like a single model with limited epistemic uncertainty across CIFAR-10/10C/SVHN and MNIST. Deep Ensembles consistently outperform BatchEnsemble in accuracy, calibration, and OOD detection, while BatchEnsemble’s diversity metrics remain near those of a single model. The work highlights the importance of reporting uncertainty diagnostics such as predictive entropy and Jensen–Shannon divergence to assess true predictive diversity and questions the practical utility of BatchEnsemble as a drop-in replacement for true ensembles in resource-constrained settings.

Abstract

In resource-constrained and low-latency settings, uncertainty estimates must be efficiently obtained. Deep Ensembles provide robust epistemic uncertainty (EU) but require training multiple full-size models. BatchEnsemble aims to deliver ensemble-like EU at far lower parameter and memory cost by applying learned rank-1 perturbations to a shared base network. We show that BatchEnsemble not only underperforms Deep Ensembles but closely tracks a single model baseline in terms of accuracy, calibration and out-of-distribution (OOD) detection on CIFAR10/10C/SVHN. A controlled study on MNIST finds members are near-identical in function and parameter space, indicating limited capacity to realize distinct predictive modes. Thus, BatchEnsemble behaves more like a single model than a true ensemble.
Paper Structure (25 sections, 15 equations, 1 figure, 7 tables)

This paper contains 25 sections, 15 equations, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Deep Ensemble.
  4. BatchEnsemble.
  5. Ensemble Diversity and EU.
  6. This work.
  7. Multiplicative Rank-1 Perturbations Have Measure Zero
  8. BatchEnsemble Underperforms Deep Ensemble
  9. BatchEnsemble Is Not Diverse
  10. Conclusion
  11. Limitations & Future Work
  12. BatchEnsemble Vectorization
  13. Proofs For Limited Expressivity Of BatchEnsemble
  14. Metrics
  15. Predictive performance.
  16. ...and 10 more sections

Figures (1)

  • Figure 1: BatchEnsemble lacks functional and parametric diversity (MNIST MLP). (a) Pairwise prediction disagreement across ID, distribution-shifted, and OOD test sets (higher is more diverse). (b) Cosine similarity of members’ weights (higher is more similar). Deep Ensembles are diverse in both spaces; BatchEnsemble members cluster tightly.

Theorems & Definitions (1)

  • proof