Decoupled Contrastive Learning for Long-Tailed Recognition

TL;DR

This paper decouples two types of positives in SCL and optimizes their relations toward different objectives to alleviate the influence of the imbalanced dataset, and proposes a patch-based self distillation to transfer knowledge from head to tail classes to relieve the under-representation of tail classes.

Abstract

Supervised Contrastive Loss (SCL) is popular in visual representation learning. Given an anchor image, SCL pulls two types of positive samples, i.e., its augmentation and other images from the same class together, while pushes negative images apart to optimize the learned embedding. In the scenario of long-tailed recognition, where the number of samples in each class is imbalanced, treating two types of positive samples equally leads to the biased optimization for intra-category distance. In addition, similarity relationship among negative samples, that are ignored by SCL, also presents meaningful semantic cues. To improve the performance on long-tailed recognition, this paper addresses those two issues of SCL by decoupling the training objective. Specifically, it decouples two types of positives in SCL and optimizes their relations toward different objectives to alleviate the influence of the imbalanced dataset. We further propose a patch-based self distillation to transfer knowledge from head to tail classes to relieve the under-representation of tail classes. It uses patch-based features to mine shared visual patterns among different instances and leverages a self distillation procedure to transfer such knowledge. Experiments on different long-tailed classification benchmarks demonstrate the superiority of our method. For instance, it achieves the 57.7% top-1 accuracy on the ImageNet-LT dataset. Combined with the ensemble-based method, the performance can be further boosted to 59.7%, which substantially outperforms many recent works. The code is available at https://github.com/SY-Xuan/DSCL.

Figures (5)

• Figure 1: Examples of retrieval results using features learned by SCL on head classes in (a) and tail classes in (b). In (b), features learned by SCL are biased to low-level appearance cues, while features learned by our method are more discriminative to semantic cues.
• Figure 2: Average ratio of gradient L2 norm computed by pulling the anchor with two types of positives as in Eq. \ref{['eq:gradient_ratio']} on ImageNet-LT. '*' denotes the theoretical ratio. SCL treats two types of positives equally, and leads to the imbalanced optimization. Two types of positives denote the data argumentation and other images in the same category.
• Figure 3: Illustration of the proposed method. Data augmentation is performed to get two global views of a training image. Then small patch is cropped from the global view. The backbone and Exponential Moving Average (EMA) backbone He2020 are used to extract normalized features. These features are used to calculate the similarity distribution with memory queue $M$. $\mathcal{L}_{dscl}$ optimizes the feature space by pulling the anchor image with its positive samples together and pushing the anchor image with its negative samples apart. $\mathcal{L}_{pbsd}$ transfers knowledge through mimicking two similarity distributions.
• Figure 4: Patch-based image retrieval results (top 3 returned) on ImageNet-LT. Query patches are highlighted with yellow bounding boxes. The response map of query patch features on the retrieved images is also illustrated.
• Figure 5: Evaluation of $\alpha$ in Eq. \ref{['eq:decouple_target']}, the number of patch-based features $L$ per anchor image, and the loss weight $\lambda$ on ImageNet-LT in (a), (b), and (c), respectively. Green dotted line in (a) denotes the baseline SCL.