Classifier Ensemble for Efficient Uncertainty Calibration of Deep Neural Networks for Image Classification

TL;DR

This work tackles the problem of unreliable uncertainty estimates in deep image classifiers by introducing an ab-initio classifier ensemble that attaches lightweight heads to a shared backbone. By training multiple heads and fusing their outputs with metamodels, the approach achieves substantial reductions in calibration errors ($$ECE$$ and $$MCE$$) while maintaining accuracy and using far fewer parameters than traditional model ensembles. The method eliminates the need for separate calibration data and demonstrates strong calibration improvements on CIFAR-100 and Tiny ImageNet across several architectures, notably with the DL and SL metamodels. The findings suggest that metamodel-based classifier ensembles offer a practical, scalable path to reliable uncertainty estimates in real-world vision systems.

Abstract

This paper investigates novel classifier ensemble techniques for uncertainty calibration applied to various deep neural networks for image classification. We evaluate both accuracy and calibration metrics, focusing on Expected Calibration Error (ECE) and Maximum Calibration Error (MCE). Our work compares different methods for building simple yet efficient classifier ensembles, including majority voting and several metamodel-based approaches. Our evaluation reveals that while state-of-the-art deep neural networks for image classification achieve high accuracy on standard datasets, they frequently suffer from significant calibration errors. Basic ensemble techniques like majority voting provide modest improvements, while metamodel-based ensembles consistently reduce ECE and MCE across all architectures. Notably, the largest of our compared metamodels demonstrate the most substantial calibration improvements, with minimal impact on accuracy. Moreover, classifier ensembles with metamodels outperform traditional model ensembles in calibration performance, while requiring significantly fewer parameters. In comparison to traditional post-hoc calibration methods, our approach removes the need for a separate calibration dataset. These findings underscore the potential of our proposed metamodel-based classifier ensembles as an efficient and effective approach to improving model calibration, thereby contributing to more reliable deep learning systems.

Figures (3)

• Figure 1: Expected Calibration Error (ECE) of ResNet50 he2016deep (left) and ResNeXt50 xie2017aggregated (right) on CIFAR-100 krizhevsky2009learning. Each model was trained with five classifier heads initialized with different random seeds but using the same backbone. The blue area represents the ECE range for the uncalibrated classifiers. Each red dot corresponds to the ECE value achieved using different ensemble techniques. The use of metamodels (SL, DL, DLL, SLpC) significantly improves the calibration performance and reduces the ECE compared to the uncalibrated baseline.
• Figure 2: The principle of the classifier ensemble involves a single backbone that feeds multiple classifiers (heads). The combination method can be freely selected.
• Figure 3: Training Process of our classifier ensemble.