Self-supervised Benchmark Lottery on ImageNet: Do Marginal Improvements Translate to Improvements on Similar Datasets?

TL;DR

The paper addresses whether marginal gains on ImageNet translate to improvements on related datasets and whether ImageNet-only benchmarks induce a benchmark lottery in SSL. It evaluates twelve SSL backbones across five ImageNet variants (ReaL, v2, Rendition, Sketch, Adversarial) to assess cross-dataset generalization, revealing that top ImageNet performers (notably DINO and SwAV) can underperform on Rendition and Sketch, while MoCo and BYOL often remain strong. Correlation analyses show high agreement between ImageNet and ReaL/v2 ($r \approx 0.99$) but substantially weaker alignment for Rendition, Sketch, and Adversarial ($r \approx 0.6$), indicating ImageNet validity as a predictor is limited under distribution shifts. The authors propose two aggregate benchmarking metrics (weighted average and geometric mean) and argue for a unified, multi-variant benchmarking approach to avoid overfitting to a single dataset, guiding more robust SSL model selection in practice.

Abstract

Machine learning (ML) research strongly relies on benchmarks in order to determine the relative effectiveness of newly proposed models. Recently, a number of prominent research effort argued that a number of models that improve the state-of-the-art by a small margin tend to do so by winning what they call a "benchmark lottery". An important benchmark in the field of machine learning and computer vision is the ImageNet where newly proposed models are often showcased based on their performance on this dataset. Given the large number of self-supervised learning (SSL) frameworks that has been proposed in the past couple of years each coming with marginal improvements on the ImageNet dataset, in this work, we evaluate whether those marginal improvements on ImageNet translate to improvements on similar datasets or not. To do so, we investigate twelve popular SSL frameworks on five ImageNet variants and discover that models that seem to perform well on ImageNet may experience significant performance declines on similar datasets. Specifically, state-of-the-art frameworks such as DINO and Swav, which are praised for their performance, exhibit substantial drops in performance while MoCo and Barlow Twins displays comparatively good results. As a result, we argue that otherwise good and desirable properties of models remain hidden when benchmarking is only performed on the ImageNet validation set, making us call for more adequate benchmarking. To avoid the "benchmark lottery" on ImageNet and to ensure a fair benchmarking process, we investigate the usage of a unified metric that takes into account the performance of models on other ImageNet variant datasets.

Figures (4)

• Figure 1: (top) Top-1 accuracy of SSL models plotted for variants of ImageNet and (bottom) changes in the top-1 accuracy ranking across the twelve evaluated models, highlighting the performance of DINO, Swav, MoCo, and Barlow TWins. Note that while DINO and Swav occupy top two spots in ImageNet validation, their performance does not translate to Rendition and Sketch datasets in which MoCo and Barlow show tremendous improvements.
• Figure 2: Example images from ImageNet variants.
• Figure 3: Comparison of top-1 linear accuracy on ImageNet Validation to five ImageNet variants. Gray lines in each figure indicate the regression lines while the shaded area depicts the confidence interval. Pearson's correlation coefficient is given on the top-left corner of each figure.
• Figure 4: Aggregate measures of accuracy for each model across all ImageNet variants. The points in the background represent the accuracy on an individual dataset.