Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?

TL;DR

This work shows that self-supervised learning can outperform supervised ImageNet training without labels by injecting inductive biases that curb reliance on spurious background cues. ReLICv2 combines unsupervised saliency masking with multi-view augmentation and an explicit invariance loss, yielding strong linear evaluation performance across diverse ResNet backbones and scale. It also demonstrates robust transfer, improved robustness and OOD generalization, and scalable performance on large unlabeled datasets like JFT-300M. Collectively, these results position ReLICv2 as a strong candidate for label-free foundation-model pretraining and motivate further integration with modern architectures such as vision transformers.

Abstract

Despite recent progress made by self-supervised methods in representation learning with residual networks, they still underperform supervised learning on the ImageNet classification benchmark, limiting their applicability in performance-critical settings. Building on prior theoretical insights from ReLIC [Mitrovic et al., 2021], we include additional inductive biases into self-supervised learning. We propose a new self-supervised representation learning method, ReLICv2, which combines an explicit invariance loss with a contrastive objective over a varied set of appropriately constructed data views to avoid learning spurious correlations and obtain more informative representations. ReLICv2 achieves $77.1\%$ top-$1$ accuracy on ImageNet under linear evaluation on a ResNet50, thus improving the previous state-of-the-art by absolute $+1.5\%$; on larger ResNet models, ReLICv2 achieves up to $80.6\%$ outperforming previous self-supervised approaches with margins up to $+2.3\%$. Most notably, ReLICv2 is the first unsupervised representation learning method to consistently outperform the supervised baseline in a like-for-like comparison over a range of ResNet architectures. Using ReLICv2, we also learn more robust and transferable representations that generalize better out-of-distribution than previous work, both on image classification and semantic segmentation. Finally, we show that despite using ResNet encoders, ReLICv2 is comparable to state-of-the-art self-supervised vision transformers.

Figures (8)

• Figure 1: Top-1 linear evaluation accuracy on ImageNet using ResNet50 encoders with $1\times$, $2\times$ and $4\times$ width multipliers and a ResNet200 encoder with a $2\times$ width multiplier.
• Figure 2: (a) ReLICv2 uses saliency masking as part of the data augmentations pipeline and views of various sizes to learn representations that are invariant to spurious correlations. Note that the $L$ (differently augmented) large views are passed through both the online and target networks, while the small $S$ views are only passed through the online network. The learning objective is computed by comparing each of the large and small views passed through the online network with each large view passed through the target network. (b) The objective used for each comparison combines the contrastive (instance discrimination) loss, i.e the cross-entropy (x-entropy) loss based on the similarity scores, and the invariance loss, i.e. the Kullback-Leibler (KL) divergence between the similarity scores across augmentations.
• Figure 3: ImageNet accuracy obtained by ReLICv2 as a function of number of images seen during pre-training for several of ResNet architectures (Number of model parameters in brackets).
• Figure 4: Distribution of the linear discriminant ratio: the ratio of between-class distances and within-class distances of embeddings computed on the ImageNet test set.
• Figure 5: Transfer performance relative to the supervised baseline (a value of 0 indicates equal performance to supervised).