Augmentations vs Algorithms: What Works in Self-Supervised Learning

TL;DR

This paper challenges the idea that self-supervised learning gains are primarily driven by algorithmic innovations. By unifying multiple SSL methods into a single framework and conducting controlled experiments on ImageNet, the authors quantify the relative impact of data augmentations, model architectures, and training losses. They find augmentation diversity and larger architectures to be the dominant factors, with momentum encoders and predictors offering modest gains and the pretext task contributing minimally to downstream performance. The results advocate prioritizing augmentation design and scale to advance SSL in practical settings, with implications for research focus and democratization of SSL benefits.

Abstract

We study the relative effects of data augmentations, pretraining algorithms, and model architectures in Self-Supervised Learning (SSL). While the recent literature in this space leaves the impression that the pretraining algorithm is of critical importance to performance, understanding its effect is complicated by the difficulty in making objective and direct comparisons between methods. We propose a new framework which unifies many seemingly disparate SSL methods into a single shared template. Using this framework, we identify aspects in which methods differ and observe that in addition to changing the pretraining algorithm, many works also use new data augmentations or more powerful model architectures. We compare several popular SSL methods using our framework and find that many algorithmic additions, such as prediction networks or new losses, have a minor impact on downstream task performance (often less than $1\%$), while enhanced augmentation techniques offer more significant performance improvements ($2-4\%$). Our findings challenge the premise that SSL is being driven primarily by algorithmic improvements, and suggest instead a bitter lesson for SSL: that augmentation diversity and data / model scale are more critical contributors to recent advances in self-supervised learning.

Figures (1)

• Figure 1: Our proposed SSL framework. The right panel shows 8 popular SSL algorithms, each of which has seemingly different model graphs, network architectures, losses, and augmentation strategies. We show that all of these methods are instances of a single unifed framework (left), each having different architecture hyperparameters for each module, augmentation hyperparameters, or losses.