Self-Soupervision: Cooking Model Soups without Labels

TL;DR

The paper addresses the limitation that traditional model soups rely on labeled data by extending the concept to self-supervised learning. It introduces Self-Soupervision, which builds soups from SSL-enabled ingredients via inter-training on unlabeled data and combines them through mixing strategies, including an unsupervised Self-Seasoning procedure. The authors demonstrate that linear mode connectivity can hold between SSL ingredients and show robustness gains on corrupted data (e.g., ImageNet-C and LAION-C) as well as transfer benefits on VTAB, with additional gains when inter-training on shifted distributions and when mixing SSL ingredients directly. By broadening the soup framework to SSL and unlabeled data, the work offers practical recipes for more robust, generalizable models without additional labeling burdens.

Abstract

Model soups are strange and strangely effective combinations of parameters. They take a model (the stock), fine-tune it into multiple models (the ingredients), and then mix their parameters back into one model (the soup) to improve predictions. While all known soups require supervised learning, and optimize the same loss on labeled data, our recipes for Self-\emph{Soup}ervision generalize soups to self-supervised learning (SSL). Our Self-Souping lets us flavor ingredients on new data sources, e.g. from unlabeled data from a task for transfer or from a shift for robustness. We show that Self-Souping on corrupted test data, then fine-tuning back on uncorrupted train data, boosts robustness by +3.5\% (ImageNet-C) and +7\% (LAION-C). Self-\emph{Soup}ervision also unlocks countless SSL algorithms to cook the diverse ingredients needed for more robust soups. We show for the first time that ingredients can differ in their SSL hyperparameters -- and more surprisingly, in their SSL algorithms. We cook soups of MAE, MoCoV3, and MMCR ingredients that are more accurate than any one single SSL ingredient.

Figures (4)

• Figure 1: Soups & Supervision. Soups fine-tune then mix models to improve predictions. The original Model Soup (left) fine-tunes across hyperparameters for the task, while Model Ratatouilles (center) first inter-trains on different labeled data then fine-tunes for the task. Both are supervised and cannot harness unlabeled data. Our Self-Soupervision (right) inter-trains across different losses and data to make more and better soups with and without labels. Our Self-Soups fine-tune then mix into a supervised model for the task. $\theta^{\text{pt}}$ is the pre-trained stock for initialization, $N$ is the number of fine-tunings on data $j$, $M$ is the number of inter-trainings on data $i$, and $\theta$ is the final model. We color supervised and self-supervised training runs / components. "ingreds." is short for model-soup ingredients.
• Figure 2: Souping across different SSL algorithms. We explore mixtures of 3 ingredient models that we inter-train with different self-supervised losses: MAE, MoCoV3, and MMCR. MAE is a pixel-reconstruction algorithm, MoCoV3 is an instance-contrastive algorithm, and MMCR is a dimension-contrastive algorithm. Through this experiment, we are the first to show that linear-mode connectivity can hold between ingredients that differ in their self-supervised training runs. The circles located at the corners of the 8 triangles represent the ingredients alone (no mix), and the inner triangles represent mixtures of the 3 ingredients. Each central inner triangle is a $(\frac{1}{3},\frac{1}{3},\frac{1}{3})$ mix, and the other inner triangles are unequal mixes (where mixture coefficients are proportional to the proximity to the corner ingredients).
• Figure 3: Model Soup mixes ingredients from a fine-tuning hyperparameter search; likewise, our Self-Soupervision mixes ingredients from a continued SSL + fine-tuning hyperparameter search. Continued SSL enables learning from unlabeled data---e.g., from the fine-tuning domain or test-set shift---before fine-tuning. Self-Souping these diverse ingredients further improves accuracy. Symbols follow Fig. \ref{['fig:soup-styles']}.
• Figure 4: Self-Souping is possible and productive for another stock: Franca.franca Mixing models independently trained using SSL (= ingredients) improves accuracy over ingredients alone. We initialize from Franca's ViT-B, inter-train using SSL, fine-tune on ImageNet, and mix: $\lambda \in 0\to1$.