Solving the long-tailed distribution problem by exploiting the synergies and balance of different techniques

TL;DR

The paper tackles long-tail recognition by integrating three techniques—Supervised Contrastive Learning (SCL), Rare-Class Sample Generator (RSG), and Label-Distribution-Aware Margin Loss (LDAM)—in a single end-to-end framework. It analyzes how SCL tightens intra-class clusters and, when combined with RSG's tail-space augmentation and LDAM's larger tail margins, yields balanced gains across head and tail classes. The authors formalize a joint loss L_total = α L_SCL + λ L_LDAM + η L_CESC + μ L_MV and show that RSG can be applied before the last ResNet block while LDAM modulates margins inversely with class frequency. Across CIFAR-10/100-LT, mini-ImageNet-LT, and ImageNet-LT, the approach achieves improved tail accuracy with minimal sacrifice to head-class performance, supported by ICD analyses that reveal expanded tail feature spaces without degrading overall class cohesion. The work demonstrates the practical value of synergistic, end-to-end long-tail learning for real-world imbalanced data scenarios, while acknowledging the computational cost of hyperparameter search for loss weights.

Abstract

In real-world data, long-tailed data distribution is common, making it challenging for models trained on empirical risk minimisation to learn and classify tail classes effectively. While many studies have sought to improve long tail recognition by altering the data distribution in the feature space and adjusting model decision boundaries, research on the synergy and corrective approach among various methods is limited. Our study delves into three long-tail recognition techniques: Supervised Contrastive Learning (SCL), Rare-Class Sample Generator (RSG), and Label-Distribution-Aware Margin Loss (LDAM). SCL enhances intra-class clusters based on feature similarity and promotes clear inter-class separability but tends to favour dominant classes only. When RSG is integrated into the model, we observed that the intra-class features further cluster towards the class centre, which demonstrates a synergistic effect together with SCL's principle of enhancing intra-class clustering. RSG generates new tail features and compensates for the tail feature space squeezed by SCL. Similarly, LDAM is known to introduce a larger margin specifically for tail classes; we demonstrate that LDAM further bolsters the model's performance on tail classes when combined with the more explicit decision boundaries achieved by SCL and RSG. Furthermore, SCL can compensate for the dominant class accuracy sacrificed by RSG and LDAM. Our research emphasises the synergy and balance among the three techniques, with each amplifying the strengths of the others and mitigating their shortcomings. Our experiment on long-tailed distribution datasets, using an end-to-end architecture, yields competitive results by enhancing tail class accuracy without compromising dominant class performance, achieving a balanced improvement across all classes.

Figures (15)

• Figure 1: (a) shows the feature distribution for long-tail recognition based on Cross-Entropy (CE), where purple triangles represent head classes and green circles represent tail classes Hybrid_contrastive. (b) Supervised Contrastive Learning (SCL) in binary classification promotes intra-class clustering and inter-class separation, which can be regarded as a mutual repulsion (represented by green arrows) between the clusters of the two distinct classes. The shadow represents the position of the feature under CE classification, and the blue arrows indicate the trajectory of feature movement. (c) Feature distribution under the influence of SCL. Ideally, where both classes are tightly clustered, a void feature region exists between them, devoid of any feature distribution, signifying a clear separation of classes. (d) Rare-Class Sample Generator (RSG) calculates a set of class centers, and for clarity, only one class center (red star) is drawn for each class. RSG drives the features of a class towards its class center. Features with a red outline further cluster towards the class center from their position under SCL (shadowed area). (e) RSG transfers the feature displacement (black arrow) between the head class samples and the class center to tail samples, generating a new tail sample (green circle with a purple outline). RSG expands the feature space of tail samples, thereby influencing the potential decision boundary. (f) Label-Distribution-Aware Margin Loss (LDAM) calculates the margin based on the number of samples of each class, providing a larger margin, $\gamma_2$, for tail samples. The solid black line represents the adjusted decision boundary by LDAM.
• Figure 2: Our model overview diagram. RSG is the abbreviation of the Rare-Class Sample Generator. Based on the experimental results from RSG, it is typically integrated before the last block of the ResNet resnet model. GAP refers to Global Average Pooling.
• Figure 3: Searching for suitable SCL and LDAM weights for the CIFAR10-LT with an Imbalance Factor of 10. The blue dashed line represents the validation accuracy of our pre-trained model over the first 80 epochs. Epochs 81 to 85 depict the process of searching for hyperparameter weights using the genetic algorithm.
• Figure 4: The Top-10 Weight Combinations for Highest Validation Accuracy on the CIFAR10-LT with an Imbalance Factor of 10. In the figure's legend, the naming convention 'LDAM_X_SCL_Y' represents the weight for LDAM loss as X and the weight for SCL loss as Y, where X and Y are specific values obtained from our experiments.
• Figure 5: The Top-10 Weight Combinations for Highest Validation Accuracy on the CIFAR10-LT with an Imbalance Factor of 50.