Understanding the Role of Functional Diversity in Weight-Ensembling with Ingredient Selection and Multidimensional Scaling

TL;DR

The paper investigates how functional diversity among ingredients drives weight-ensembling performance by introducing two novel algorithms, greedier and ranked, and a distance-based framework to analyze ingredient selection. It defines diversity metrics, including the ratio-error distance $d_D$ and Euclidean distance $d_E$, to study how selections influence WA performance on ID and OOD tasks within a DomainBed OfficeHome setting. Empirical results show greedier often yields faster ID gains and better OOD accuracy than greedy and ranked variants, while maximal diversity alone does not guarantee peak performance; diversity helps but must be leveraged effectively. A qualitative Multidimensional Scaling (MDS) visualization demonstrates that successful WA configurations traverse distinct regions of weight space, linking diversity, loss-landscape structure, and generalization.

Abstract

Weight-ensembles are formed when the parameters of multiple neural networks are directly averaged into a single model. They have demonstrated generalization capability in-distribution (ID) and out-of-distribution (OOD) which is not completely understood, though they are thought to successfully exploit functional diversity allotted by each distinct model. Given a collection of models, it is also unclear which combination leads to the optimal weight-ensemble; the SOTA is a linear-time ``greedy" method. We introduce two novel weight-ensembling approaches to study the link between performance dynamics and the nature of how each method decides to use apply the functionally diverse components, akin to diversity-encouragement in the prediction-ensemble literature. We develop a visualization tool to explain how each algorithm explores various domains defined via pairwise-distances to further investigate selection and algorithms' convergence. Empirical analyses shed perspectives which reinforce how high-diversity enhances weight-ensembling while qualifying the extent to which diversity alone improves accuracy. We also demonstrate that sampling positionally distinct models can contribute just as meaningfully to improvements in a weight-ensemble.

Figures (16)

• Figure 1: Difference between greedier accuracy and other methods' accuracy averaged across all trials with 95% confidence interval. Training at left, testing at right. Terminal value carried forward.
• Figure 2: Box-plot of quantiles of diversity distance between the current WA and the selected model at each iteration $t$ of each algorithm across the 40 trials. Dashed red-line at 50% indicates random selection.
• Figure 3: $t$-incorrect ingredient-correct: Probabilities that the next-step WA predicts correctly given that the current-step WA was incorrect and the ingredient was correct, difference from greedier and other methods' averaged across all trials with 95% confidence interval. Training at left, testing at right. Terminal value carried forward.
• Figure 4: Box-plot of diversity distance between the current WA and the selected model at each iteration $t$ of each algorithm across the 40 trials.
• Figure 5: Diversity distance between the current WA and the selected model at each iteration $t$ of the greedy, greedier, ranked-diveristy, and ranked-Euclidean algorithms averaged across all trials with 95% confidence interval. First ten iterations plotted.