Can Generative Models Improve Self-Supervised Representation Learning?

TL;DR

This paper addresses the limited diversity of traditional SSL augmentations by introducing instance-conditioned generative augmentations that preserve semantic content while expanding visual variation. By integrating conditional generators (Stable Diffusion and ICGAN) with existing joint-embedding SSL methods (e.g., SimCLR, BYOL, MoCo, SimSiam, Barlow Twins) and using offline generated samples, the approach yields consistent improvements in downstream linear-probing accuracy on ImageNet and other datasets, with gains up to about 10%. The study includes a dissimilarity analysis (CKA, OPD) showing that the learned representations differ from those of CLIP, confirming that the augmented SSL space is not merely replicating pretrained encodings. The work highlights practical benefits of synthetic data for SSL while outlining future directions such as co-training the generator with the SSL model and addressing ethical considerations related to biases in generative models.

Abstract

The rapid advancement in self-supervised representation learning has highlighted its potential to leverage unlabeled data for learning rich visual representations. However, the existing techniques, particularly those employing different augmentations of the same image, often rely on a limited set of simple transformations that cannot fully capture variations in the real world. This constrains the diversity and quality of samples, which leads to sub-optimal representations. In this paper, we introduce a framework that enriches the self-supervised learning (SSL) paradigm by utilizing generative models to produce semantically consistent image augmentations. By directly conditioning generative models on a source image, our method enables the generation of diverse augmentations while maintaining the semantics of the source image, thus offering a richer set of data for SSL. Our extensive experimental results on various joint-embedding SSL techniques demonstrate that our framework significantly enhances the quality of learned visual representations by up to 10\% Top-1 accuracy in downstream tasks. This research demonstrates that incorporating generative models into the joint-embedding SSL workflow opens new avenues for exploring the potential of synthetic data. This development paves the way for more robust and versatile representation learning techniques.

Figures (7)

• Figure 1: Generative augmentations produce a more diverse set of images with similar semantics. a) The standard SSL augmentations offer limited diversity for effective representation learning. b) By generating instance-conditioned samples, and then applying the standard augmentations on top, we add more diversity in training data, leading to better representations.
• Figure 2: Our augmentation pipeline utilizes generative models, i.e., Stable Diffusion or ICGAN, conditioned on the source image representation, accompanied by the standard SSL augmentations. The components inside the Generative Augmentation module, i.e. the pretrained SSL encoder and the generative model remain frozen throughout the SSL training process.
• Figure 3: Examples of various augmentations. Compared to the standard augmentations (second row), instance-based generative augmentations can produce more diverse and realistic images that preserve the semantics of the original image.
• Figure 4: The effect of different probability values of applying the generative augmentation.
• Figure 5: Top-1 accuracy improvement on ImageNet validation set obtained by the generative augmentations with Stable Diffusion and ICGAN across five SSL techniques.