Long-tailed Recognition with Model Rebalancing

TL;DR

MORE tackles long-tailed recognition by reallocating model capacity from majority to minority classes through a low-rank tail component and a discrepancy-based loss with a sinusoidal training schedule. The approach yields a tighter balanced generalization bound and maintains identical inference cost while enhancing tail performance across both single-label and multi-label tasks, including CLIP-finetuned scenarios. Empirical results on diverse benchmarks demonstrate consistent gains for Few/medium classes without sacrificing head-class performance, and ablations validate the importance of the rebalancing loss, schedule, and discrepancy metric. This work provides a practical, theory-grounded module for improving imbalanced learning in real-world vision systems with broad applicability and potential extensions to other modalities.

Abstract

Long-tailed recognition is ubiquitous and challenging in deep learning and even in the downstream finetuning of foundation models, since the skew class distribution generally prevents the model generalization to the tail classes. Despite the promise of previous methods from the perspectives of data augmentation, loss rebalancing and decoupled training etc., consistent improvement in the broad scenarios like multi-label long-tailed recognition is difficult. In this study, we dive into the essential model capacity impact under long-tailed context, and propose a novel framework, Model Rebalancing (MORE), which mitigates imbalance by directly rebalancing the model's parameter space. Specifically, MORE introduces a low-rank parameter component to mediate the parameter space allocation guided by a tailored loss and sinusoidal reweighting schedule, but without increasing the overall model complexity or inference costs. Extensive experiments on diverse long-tailed benchmarks, spanning multi-class and multi-label tasks, demonstrate that MORE significantly improves generalization, particularly for tail classes, and effectively complements existing imbalance mitigation methods. These results highlight MORE's potential as a robust plug-and-play module in long-tailed settings.

Figures (4)

• Figure 1: An overview of the proposed method's framework. The left figure illustrates how our model rebalancing is designed. The right figure presents the performance on the NUS-WIDE-SCENE dataset across the Many/Medium/Few splits. Our method demonstrates a significant improvement in the performance of minority classes, while maintaining or enhancing the performance of other classes.
• Figure 2: (a, b) mAP (%) ($\uparrow$) at 112$\times$112 and 224$\times$224 resolutions, on MIML and NUS-WIDE-SCENE, respectively. (c) The impact of the peak amplitude $A$ in MORE. (d) The impact of the rank $r$ in MORE. In (c) and (d), experiments are conducted on MIML combined with BCE.
• Figure 3: (a,b) mAP (%) ($\uparrow$) using KL divergence and $\ell_2$ distance (MORE) as the discrepancy measure, with and without CLIP, respectively. (c,d) mAP (%) ($\uparrow$) of the baseline model, MORE w/o ${\mathcal{L}}_{\mathrm{MORE}}$, compared to MORE, on Pascal-VOC and MS-COCO, respectively.
• Figure 4: (a) Decomposition of hypothesis space $\mathcal{G}$. (b) Empirical risk comparison between baseline and MORE. Experiments are conducted on MIML using BCE as base loss.

Theorems & Definitions (2)

• Theorem 1
• Lemma 1