Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Weak Augmentation Guided Relational Self-Supervised Learning

Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, Chang Xu

TL;DR

This work introduces Relational Self-Supervised Learning (ReSSL), a framework that preserves inter-instance relations rather than enforcing strict instance discrimination. By computing sharpened relation distributions from pairwise similarities with a momentum-updated memory bank and employing weak teacher augmentations, ReSSL provides stable relational targets and strong representation quality. The approach is enhanced with an asymmetric predictor and an InfoNCE warm-up, yielding improved performance on ImageNet and strong results for lightweight architectures, often outperforming prior KD and contrastive methods at reduced training cost. Overall, ReSSL demonstrates that relational consistency is a powerful and practical principle for unsupervised visual representation learning with broad applicability to both large-scale and compact models.

Abstract

Self-supervised Learning (SSL) including the mainstream contrastive learning has achieved great success in learning visual representations without data annotations. However, most methods mainly focus on the instance level information (\ie, the different augmented images of the same instance should have the same feature or cluster into the same class), but there is a lack of attention on the relationships between different instances. In this paper, we introduce a novel SSL paradigm, which we term as relational self-supervised learning (ReSSL) framework that learns representations by modeling the relationship between different instances. Specifically, our proposed method employs sharpened distribution of pairwise similarities among different instances as \textit{relation} metric, which is thus utilized to match the feature embeddings of different augmentations. To boost the performance, we argue that weak augmentations matter to represent a more reliable relation, and leverage momentum strategy for practical efficiency. The designed asymmetric predictor head and an InfoNCE warm-up strategy enhance the robustness to hyper-parameters and benefit the resulting performance. Experimental results show that our proposed ReSSL substantially outperforms the state-of-the-art methods across different network architectures, including various lightweight networks (\eg, EfficientNet and MobileNet).

Weak Augmentation Guided Relational Self-Supervised Learning

TL;DR

This work introduces Relational Self-Supervised Learning (ReSSL), a framework that preserves inter-instance relations rather than enforcing strict instance discrimination. By computing sharpened relation distributions from pairwise similarities with a momentum-updated memory bank and employing weak teacher augmentations, ReSSL provides stable relational targets and strong representation quality. The approach is enhanced with an asymmetric predictor and an InfoNCE warm-up, yielding improved performance on ImageNet and strong results for lightweight architectures, often outperforming prior KD and contrastive methods at reduced training cost. Overall, ReSSL demonstrates that relational consistency is a powerful and practical principle for unsupervised visual representation learning with broad applicability to both large-scale and compact models.

Abstract

Self-supervised Learning (SSL) including the mainstream contrastive learning has achieved great success in learning visual representations without data annotations. However, most methods mainly focus on the instance level information (\ie, the different augmented images of the same instance should have the same feature or cluster into the same class), but there is a lack of attention on the relationships between different instances. In this paper, we introduce a novel SSL paradigm, which we term as relational self-supervised learning (ReSSL) framework that learns representations by modeling the relationship between different instances. Specifically, our proposed method employs sharpened distribution of pairwise similarities among different instances as \textit{relation} metric, which is thus utilized to match the feature embeddings of different augmentations. To boost the performance, we argue that weak augmentations matter to represent a more reliable relation, and leverage momentum strategy for practical efficiency. The designed asymmetric predictor head and an InfoNCE warm-up strategy enhance the robustness to hyper-parameters and benefit the resulting performance. Experimental results show that our proposed ReSSL substantially outperforms the state-of-the-art methods across different network architectures, including various lightweight networks (\eg, EfficientNet and MobileNet).
Paper Structure (20 sections, 7 equations, 3 figures, 20 tables)

This paper contains 20 sections, 7 equations, 3 figures, 20 tables.

Table of Contents

  1. Introduction
  2. Introduction
  3. Related Works
  4. Methodology
  5. Preliminaries on Self-supervised Learning
  6. Relational Self-Supervised Learning
  7. Improving ReSSL with Predictor and InfoNCE
  8. Empirical Study of ReSSL
  9. A Properly Sharpened Relation is A Better Target
  10. Asymmetric Structure Makes Robuster System
  11. Working with InfoNCE Warm-up
  12. Weak Augmentation Makes Better Relation
  13. More Experiments on Weak Augmentation
  14. Dimension of the Relation
  15. Visualization of Learned Representations
  16. ...and 5 more sections

Figures (3)

  • Figure 1: The overall framework of our proposed method. We adopt the student-teacher framework where the student is trained to predict the representation of the teacher, and the teacher is updated with a “momentum update” (exponential moving average) of the student. The relationship consistency is achieve by align the conditional distribution for student and teacher model. Please see more details in our method part.
  • Figure 2: Visualization of the 10 nearest neighbour of the query image. The top half is the result when we apply the weak augmentation. The bottom half is the case when the typical contrastive augmentation is adopted. Note, we use the red square box to highlight the images that has different ground truth label with the query image.
  • Figure 3: t-SNE visualizations on CIFAR-10. Classes are indicated by colors. Here we show the visualization result for ReSSL with contrastive augmentation, Standard ReSSL with weak augmentation, and MoCo V2.