Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dataset Awareness is not Enough: Implementing Sample-level Tail Encouragement in Long-tailed Self-supervised Learning

Haowen Xiao, Guanghui Liu, Xinyi Gao, Yang Li, Fengmao Lv, Jielei Chu

TL;DR

This work introduces pseudo-labels into self-supervised long-tailed learning, utilizing pseudo-label information to drive a dynamic temperature and re-weighting strategy and achieves outstanding performance in improving long-tail recognition, while also exhibiting high robustness.

Abstract

Self-supervised learning (SSL) has shown remarkable data representation capabilities across a wide range of datasets. However, when applied to real-world datasets with long-tailed distributions, performance on multiple downstream tasks degrades significantly. Recently, the community has begun to focus more on self-supervised long-tailed learning. Some works attempt to transfer temperature mechanisms to self-supervised learning or use category-space uniformity constraints to balance the representation of different categories in the embedding space to fight against long-tail distributions. However, most of these approaches focus on the joint optimization of all samples in the dataset or on constraining the category distribution, with little attention given to whether each individual sample is optimally guided during training. To address this issue, we propose Temperature Auxiliary Sample-level Encouragement (TASE). We introduce pseudo-labels into self-supervised long-tailed learning, utilizing pseudo-label information to drive a dynamic temperature and re-weighting strategy. Specifically, We assign an optimal temperature parameter to each sample. Additionally, we analyze the lack of quantity awareness in the temperature parameter and use re-weighting to compensate for this deficiency, thereby achieving optimal training patterns at the sample level. Comprehensive experimental results on six benchmarks across three datasets demonstrate that our method achieves outstanding performance in improving long-tail recognition, while also exhibiting high robustness.

Dataset Awareness is not Enough: Implementing Sample-level Tail Encouragement in Long-tailed Self-supervised Learning

TL;DR

This work introduces pseudo-labels into self-supervised long-tailed learning, utilizing pseudo-label information to drive a dynamic temperature and re-weighting strategy and achieves outstanding performance in improving long-tail recognition, while also exhibiting high robustness.

Abstract

Self-supervised learning (SSL) has shown remarkable data representation capabilities across a wide range of datasets. However, when applied to real-world datasets with long-tailed distributions, performance on multiple downstream tasks degrades significantly. Recently, the community has begun to focus more on self-supervised long-tailed learning. Some works attempt to transfer temperature mechanisms to self-supervised learning or use category-space uniformity constraints to balance the representation of different categories in the embedding space to fight against long-tail distributions. However, most of these approaches focus on the joint optimization of all samples in the dataset or on constraining the category distribution, with little attention given to whether each individual sample is optimally guided during training. To address this issue, we propose Temperature Auxiliary Sample-level Encouragement (TASE). We introduce pseudo-labels into self-supervised long-tailed learning, utilizing pseudo-label information to drive a dynamic temperature and re-weighting strategy. Specifically, We assign an optimal temperature parameter to each sample. Additionally, we analyze the lack of quantity awareness in the temperature parameter and use re-weighting to compensate for this deficiency, thereby achieving optimal training patterns at the sample level. Comprehensive experimental results on six benchmarks across three datasets demonstrate that our method achieves outstanding performance in improving long-tail recognition, while also exhibiting high robustness.

Paper Structure

This paper contains 23 sections, 6 equations, 3 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Preliminary
  5. Temperature Property
  6. Structural Separability and Category Uniformity
  7. Explicit Quantity Awareness
  8. Training via TASE
  9. More Discussion
  10. Experiment
  11. Datasets
  12. Experimental Setup
  13. Experimental Results
  14. Analysis of Sensitivity
  15. Ablation Study
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Comparison between SimCLR SimCLR baseline trained on balanced CIFAR10cifar and imbalanced CIFAR10-LT cifar_LT and our TASE trained on CIFAR10-LT. Three Splits (Head, Mid and Tail) are shown as different colours (blue, red and green) respectively liu2019large. We also report the average accuracy and range of variation over ten classes.
  • Figure 2: Illustration of the structure of the desired embedding space. We use different icons in different colors to represent the samples from different categories and indicate the size of the space occupied by each category. We expect the clusters to be at a certain distance from each other and to occupy a similar amount of space.
  • Figure 3: T-SNE Visualization. Different colors indicate different classes.