Long-Tail Learning with Rebalanced Contrastive Loss

TL;DR

This work tackles the long-tail learning problem by enhancing supervised contrastive learning through Rebalanced Contrastive Learning (RCL). RCL introduces class-frequency based balancing (Balanced SoftMax), feature-space balancing, intra-class compactness via feature scaling, and embedding-margin regularization to improve tail-class discrimination, implemented on the Balanced Contrastive Learning (BCL) framework and also effective as a standalone loss. Experimental results on CIFAR10_Lt, CIFAR100_Lt, and ImageNet_Lt demonstrate improved top-1 balanced accuracy and better embedding space properties, with ablations confirming the contribution of each component. The approach provides a simple, compatible enhancement to existing SCL methods that boosts tail-class performance while preserving head-class accuracy, with potential for extension to self-supervised variants.

Abstract

Integrating supervised contrastive loss to cross entropy-based communication has recently been proposed as a solution to address the long-tail learning problem. However, when the class imbalance ratio is high, it requires adjusting the supervised contrastive loss to support the tail classes, as the conventional contrastive learning is biased towards head classes by default. To this end, we present Rebalanced Contrastive Learning (RCL), an efficient means to increase the long tail classification accuracy by addressing three main aspects: 1. Feature space balancedness - Equal division of the feature space among all the classes, 2. Intra-Class compactness - Reducing the distance between same-class embeddings, 3. Regularization - Enforcing larger margins for tail classes to reduce overfitting. RCL adopts class frequency-based SoftMax loss balancing to supervised contrastive learning loss and exploits scalar multiplied features fed to the contrastive learning loss to enforce compactness. We implement RCL on the Balanced Contrastive Learning (BCL) Framework, which has the SOTA performance. Our experiments on three benchmark datasets demonstrate the richness of the learnt embeddings and increased top-1 balanced accuracy RCL provides to the BCL framework. We further demonstrate that the performance of RCL as a standalone loss also achieves state-of-the-art level accuracy.

Figures (3)

• Figure 1: (a) This figure denotes the distribution of the penultimate layer embeddings in a 2D space in the class imbalanced case. Important factors to note are 1) feature distribution in the hyperspace, 2) class margins, and 3) feature deviation in each class. (b) Shows the expected ideal feature distribution under class imbalance. Symmetrically distributed class centres in the hyperspace, feature compactness, clear separability and generalisation of the class margins need to be addressed by the SCL components.
• Figure 2: RCL can be implemented with parallel branches as in the work of bcl for classifier learning and supervised contrastive learning, as indicated in the figure. $v_1$ is the original image and $v_2$ and $v_3$ are augmentations of it. We adopt the same augmentation methods in BCL framework. A common backbone is used for both branches. Total loss is minimized during the training process.
• Figure 3: Learnt embedding distribution for CIFAR10_Lt and CIFAR100_Lt (IF=100:1) in TSNE format. Results showcase that the RCL addition has contributed to more balanced and separable tail class embeddings.