Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance

TL;DR

The paper addresses the challenge that reducing per-model sharpness in deep ensembles via sharpness-aware optimization can unintentionally diminish ensemble diversity, limiting gains in performance. It provides theoretical and empirical evidence for a sharpness-diversity trade-off and introduces SharpBalance, a data-subset training strategy where each ensemble member minimizes sharpness on a distinct subset, improving diversity for a given sharpness level. The approach is backed by theoretical bounds and validated on CIFAR-10/100 and TinyImageNet, showing improved ID and OOD generalization with only minor training overhead. Overall, the work advances ensemble robustness by balancing local minima flatness with inter-model diversity, enabling more reliable uncertainty quantification and generalization to out-of-distribution data.

Abstract

Recent studies on deep ensembles have identified the sharpness of the local minima of individual learners and the diversity of the ensemble members as key factors in improving test-time performance. Building on this, our study investigates the interplay between sharpness and diversity within deep ensembles, illustrating their crucial role in robust generalization to both in-distribution (ID) and out-of-distribution (OOD) data. We discover a trade-off between sharpness and diversity: minimizing the sharpness in the loss landscape tends to diminish the diversity of individual members within the ensemble, adversely affecting the ensemble's improvement. The trade-off is justified through our theoretical analysis and verified empirically through extensive experiments. To address the issue of reduced diversity, we introduce SharpBalance, a novel training approach that balances sharpness and diversity within ensembles. Theoretically, we show that our training strategy achieves a better sharpness-diversity trade-off. Empirically, we conducted comprehensive evaluations in various data sets (CIFAR-10, CIFAR-100, TinyImageNet) and showed that SharpBalance not only effectively improves the sharpness-diversity trade-off, but also significantly improves ensemble performance in ID and OOD scenarios.

Figures (13)

• Figure 1: (Sharpness-diversity trade-off and SharpBalance).(a) Caricature illustrating the sharpness-diversity trade-off that emerges in an ensemble's loss landscape induced by the Sharpness-aware Minimization (SAM) optimizer. We propose SharpBalance to address this trade-off. Each black circle represents an individual NN in a three-member ensemble. The distance between circles represents the diversity between NNs and the ruggedness of the basin represents the sharpness of each NN. (b) Theoretically proving the existence of the sharpness-diversity trade-off and improvement from SharpBalance, plotting the analytic representation of sharpness and diversity from Theorem \ref{['thm:varSAM']} and Theorem \ref{['thm:sub']} by changing the perturbation radius $\rho$ of SAM. SharpBalance achieves a larger diversity for the same level of sharpness. (c) Empirical results of verifying sharpness-diversity trade-off improvement from SharpBalance. Each marker represents a three-member ResNet18 ensemble trained on CIFAR-10. Diversity is measured by the variance of individual models' predictions, and sharpness is measured by the adaptive worst-case sharpness, both defined in Section \ref{['sec:notation']}.
• Figure 2: (Theoretical vs. Simulated sharpness-diversity trade-off). This figure illustrates the relationship between sharpness (upper and lower bounds) and diversity as predicted by Thereom \ref{['thm:varSAM']} and as observed in simulations. Note that the upper and lower bounds correspond to the sharpness values plotted along the x-axis, with the upper bound positioned to the right and the lower bound to the left. Also, note that the bounds provided are for the expected sharpness, which means that random fluctuations can cause the simulation results to move beyond these bounds.
• Figure 3: (Varying diversity measure in empirical study). Three different metrics are employed to measure the diversity of individual models within an ensemble, i.e., Variance in equation (\ref{['eq:defdiv']}), DER in equation (\ref{['eqn:disagreement']}), and KL divergence in equation (\ref{['eqn:kl-divergence']}). The results of the three metrics show consistent trends, demonstrating the sharpness-diversity trade-off: lower sharpness is correlated with lower diversity. The experiment is conducted by training a three-member ResNet18 ensemble on CIFAR10.
• Figure 4: (Empirical observations of sharpness-diversity trade-off). The identified trade-off shows that while reducing sharpness enhances individual model performance, it concurrently lowers diversity and thus diminishes the ensemble improvement rate. First row: the color encoding represents the ensemble improvement rate (EIR) defined in equation (\ref{['eqn:eir']}), from red to blue means ensembling improvement decreases. Second row: the color encoding represents the individual ensemble member's OOD accuracy, from blue to red means individual performance becomes better. Each marker represents a three-member ResNet18 ensemble trained with SAM with a different perturbation radius.
• Figure 5: (Sharpness-diversity trade-off in models varying overparameterization levels). Different types of markers represent models with varying degrees of overparameterization, determined by changing the model width (a) or sparsity (b). Each marker represents a three-member ensemble trained with SAM with a different perturbation radius. The $\beta$ reflects the rate of decline in the trade-off curve, calculated via applying linear fitting over the ensembles at each level of overparameterization. A higher $\beta$ points to a steeper decline in the trade-off. Ensembles with narrower widths or increased sparsity display more pronounced trade-off effects. The model used in ResNet18 and the dataset is CIFAR-10.

Theorems & Definitions (8)

• Theorem 1: Diversity and Sharpness of SAM
• Theorem 2: Diversity and Sharpness when Models are Trained on Subsets
• Theorem 3: Unrolling SAM
• proof
• Proposition 1: Expectation of Wishart Moments
• proof
• proof
• proof