Difficulty-aware Balancing Margin Loss for Long-tailed Recognition

TL;DR

This work tackles long-tailed recognition by introducing the Difficulty-aware Balancing Margin (DBM) loss, which jointly uses a class-wise margin $m_C$ to compensate for imbalance and an instance-wise margin $m_I$ to emphasize hard positives, all within a cosine classifier framework. The class-wise margin scales as $m_C = K \rho_y^{-\tau}$, where $\rho_y$ is the class frequency ratio, and the instance-wise margin scales with sample difficulty via $m_I = m_C d_I$ with $d_I = (1 - \cos\theta_y)/2$, applied only to hard positives. DBM can be plugged into existing LTR methods (e.g., CE, BS, LDAM, BCL, GML, NCL) with minimal overhead, and experiments across CIFAR-10/100-LT, ImageNet-LT, and iNaturalist2018 show improved tail-class performance and robust gains in intra-class compactness and inter-class separability. The results demonstrate that jointly addressing class imbalance and within-class instance difficulty yields more discriminative features and better generalization on highly skewed datasets. Overall, DBM provides a practical and versatile enhancement for long-tailed recognition that strengthens performance where it matters most—tail classes and hard instances.

Abstract

When trained with severely imbalanced data, deep neural networks often struggle to accurately recognize classes with only a few samples. Previous studies in long-tailed recognition have attempted to rebalance biased learning using known sample distributions, primarily addressing different classification difficulties at the class level. However, these approaches often overlook the instance difficulty variation within each class. In this paper, we propose a difficulty-aware balancing margin (DBM) loss, which considers both class imbalance and instance difficulty. DBM loss comprises two components: a class-wise margin to mitigate learning bias caused by imbalanced class frequencies, and an instance-wise margin assigned to hard positive samples based on their individual difficulty. DBM loss improves class discriminativity by assigning larger margins to more difficult samples. Our method seamlessly combines with existing approaches and consistently improves performance across various long-tailed recognition benchmarks.

Figures (4)

• Figure 1: Overview of our method. The model is trained to align samples within decision boundaries defined by adaptive margins. (1) Hard positive samples. Misclassified samples identified during training are labeled as hard positive samples. (2) Class-wise margins. Larger margins are assigned to minority classes to ensure sufficient separation from majority classes. (3) Instance-wise Margins. We propose to apply adaptive margins to hard positive samples, considering both class frequency and sample difficulty.
• Figure 2: Margins for $K=0.1$ and $\tau=1$. Less frequent classes have larger class-wise margin, and more difficult samples have larger instance-wise margin.
• Figure 3: Analysis for effects of hyperparameters $\tau$ and $K$. For all cases, DBM-BS outperforms the baseline BS bs_ren performance of 49.35%.
• Figure 4: Comparison of BS bs_ren and DBM-BS of the distribution of angular distance between sample features and their positive class centers for 'Many', 'Medium', and 'Few' groups. Dashed horizontal lines denote the quartiles.