Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity

TL;DR

FreeTickets tackles the high cost of traditional deep ensembles by leveraging Dynamic Sparse Training (DST) to construct an ensemble from many cheap sparse subnetworks. It introduces two concrete instantiations, DST Ensemble and EDST Ensemble, which produce diverse yet accurate tickets with significantly reduced training and inference costs, even achieving near-dense-ensemble performance on large-scale data with as little as $0.57$–$0.61$ of the training FLOPs. Empirically, FreeTickets improves accuracy, uncertainty estimation, and OoD robustness compared to dense baselines and other efficient ensembles, while reducing resource usage; the diversity among subnetworks plays a crucial role in these gains. This work demonstrates that sparsity, when orchestrated via DST, not only preserves expressibility but also yields richer ensemble behavior, offering practical speedups and new directions for hardware-aware sparse learning.

Abstract

The success of deep ensembles on improving predictive performance, uncertainty estimation, and out-of-distribution robustness has been extensively studied in the machine learning literature. Albeit the promising results, naively training multiple deep neural networks and combining their predictions at inference leads to prohibitive computational costs and memory requirements. Recently proposed efficient ensemble approaches reach the performance of the traditional deep ensembles with significantly lower costs. However, the training resources required by these approaches are still at least the same as training a single dense model. In this work, we draw a unique connection between sparse neural network training and deep ensembles, yielding a novel efficient ensemble learning framework called FreeTickets. Instead of training multiple dense networks and averaging them, we directly train sparse subnetworks from scratch and extract diverse yet accurate subnetworks during this efficient, sparse-to-sparse training. Our framework, FreeTickets, is defined as the ensemble of these relatively cheap sparse subnetworks. Despite being an ensemble method, FreeTickets has even fewer parameters and training FLOPs than a single dense model. This seemingly counter-intuitive outcome is due to the ultra training/inference efficiency of dynamic sparse training. FreeTickets surpasses the dense baseline in all the following criteria: prediction accuracy, uncertainty estimation, out-of-distribution (OoD) robustness, as well as efficiency for both training and inference. Impressively, FreeTickets outperforms the naive deep ensemble with ResNet50 on ImageNet using around only 1/5 of the training FLOPs required by the latter. We have released our source code at https://github.com/VITA-Group/FreeTickets.

Figures (7)

• Figure 1: Illustration of $FreeTickets$ with EDST Ensemble as an example. EDST Ensemble, consisting of one exploration phase and $\mathrm{M}$ sequential refinement phases, produces $\mathrm{M}$ diverse subnetworks with very low cost (hence called "free tickets"). By combining all these free tickets, EDST Ensemble matches the performance of the dense ensemble with only half of FLOPs required to train a single dense model.
• Figure 2: Left: KL divergence across intermediate feature maps of various sparse ensemble methods. Each line is the averaged KL divergence among $\mathrm{M=3}$ subnetworks. Right: Diversity comparison between sparse ensemble methods and non-sparse ensemble methods with Wide ResNet28-10 on CIFAR-10 ($\mathrm{M}=3$, $\mathrm{S}=0.8$).
• Figure 3: t-SNE projection of training trajectories of ensemble learners discovered by DST Ensemble and EDST Ensemble with Wide ResNet28-10 on CIFAR-10/100. The sparsity level is $\mathrm{S}=0.8$.
• Figure 4: Performance and KL divergence of the subnetworks and the ensemble of subnetworks as sparsity varies. The KL divergence is scaled to the test accuracy for better visualization.
• Figure 5: Performance of the subnetworks and the ensemble of subnetworks as the ensemble size $\mathrm{M}$ varies (Wide ResNet28-10 on CIFAR-10).