Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Invariant Feature Learning for Generalized Long-Tailed Classification

Kaihua Tang, Mingyuan Tao, Jiaxin Qi, Zhenguang Liu, Hanwang Zhang

TL;DR

This work introduces Generalized Long-Tailed Classification (GLT), which jointly tackles class-wise and attribute-wise imbalances that arise in real-world data. It formalizes GLT via a Bayes decomposition that separates invariant class factors $z_c$ from varying attributes $z_a$, and proposes ImageNet-GLT and MSCOCO-GLT benchmarks with CLT, ALT, and GLT protocols to test robustness under both biases. The proposed Invariant Feature Learning (IFL) backbone constructs diverse environments and uses an invariant risk minimization–style metric loss $L_{IFL}$ to encourage class features that are robust to attributes, demonstrating improvements across standard LT methods when evaluated on GLT benchmarks. The findings indicate that learning attribute-invariant representations can simultaneously improve precision and accuracy, reduce spurious correlations, and provide a strong, broadly applicable enhancement to LT approaches. The authors also release benchmarks and code to support further research in GLT.

Abstract

Existing long-tailed classification (LT) methods only focus on tackling the class-wise imbalance that head classes have more samples than tail classes, but overlook the attribute-wise imbalance. In fact, even if the class is balanced, samples within each class may still be long-tailed due to the varying attributes. Note that the latter is fundamentally more ubiquitous and challenging than the former because attributes are not just implicit for most datasets, but also combinatorially complex, thus prohibitively expensive to be balanced. Therefore, we introduce a novel research problem: Generalized Long-Tailed classification (GLT), to jointly consider both kinds of imbalances. By "generalized", we mean that a GLT method should naturally solve the traditional LT, but not vice versa. Not surprisingly, we find that most class-wise LT methods degenerate in our proposed two benchmarks: ImageNet-GLT and MSCOCO-GLT. We argue that it is because they over-emphasize the adjustment of class distribution while neglecting to learn attribute-invariant features. To this end, we propose an Invariant Feature Learning (IFL) method as the first strong baseline for GLT. IFL first discovers environments with divergent intra-class distributions from the imperfect predictions and then learns invariant features across them. Promisingly, as an improved feature backbone, IFL boosts all the LT line-up: one/two-stage re-balance, augmentation, and ensemble. Codes and benchmarks are available on Github: https://github.com/KaihuaTang/Generalized-Long-Tailed-Benchmarks.pytorch

Invariant Feature Learning for Generalized Long-Tailed Classification

TL;DR

This work introduces Generalized Long-Tailed Classification (GLT), which jointly tackles class-wise and attribute-wise imbalances that arise in real-world data. It formalizes GLT via a Bayes decomposition that separates invariant class factors from varying attributes , and proposes ImageNet-GLT and MSCOCO-GLT benchmarks with CLT, ALT, and GLT protocols to test robustness under both biases. The proposed Invariant Feature Learning (IFL) backbone constructs diverse environments and uses an invariant risk minimization–style metric loss to encourage class features that are robust to attributes, demonstrating improvements across standard LT methods when evaluated on GLT benchmarks. The findings indicate that learning attribute-invariant representations can simultaneously improve precision and accuracy, reduce spurious correlations, and provide a strong, broadly applicable enhancement to LT approaches. The authors also release benchmarks and code to support further research in GLT.

Abstract

Existing long-tailed classification (LT) methods only focus on tackling the class-wise imbalance that head classes have more samples than tail classes, but overlook the attribute-wise imbalance. In fact, even if the class is balanced, samples within each class may still be long-tailed due to the varying attributes. Note that the latter is fundamentally more ubiquitous and challenging than the former because attributes are not just implicit for most datasets, but also combinatorially complex, thus prohibitively expensive to be balanced. Therefore, we introduce a novel research problem: Generalized Long-Tailed classification (GLT), to jointly consider both kinds of imbalances. By "generalized", we mean that a GLT method should naturally solve the traditional LT, but not vice versa. Not surprisingly, we find that most class-wise LT methods degenerate in our proposed two benchmarks: ImageNet-GLT and MSCOCO-GLT. We argue that it is because they over-emphasize the adjustment of class distribution while neglecting to learn attribute-invariant features. To this end, we propose an Invariant Feature Learning (IFL) method as the first strong baseline for GLT. IFL first discovers environments with divergent intra-class distributions from the imperfect predictions and then learns invariant features across them. Promisingly, as an improved feature backbone, IFL boosts all the LT line-up: one/two-stage re-balance, augmentation, and ensemble. Codes and benchmarks are available on Github: https://github.com/KaihuaTang/Generalized-Long-Tailed-Benchmarks.pytorch
Paper Structure (27 sections, 5 equations, 10 figures, 6 tables, 2 algorithms)

This paper contains 27 sections, 5 equations, 10 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Generalized Long-Tailed Classification
  4. Problem Formulation
  5. GLT Benchmarks and Evaluation Protocols
  6. Invariant Feature Learning
  7. Environment Construction
  8. IFL Metric Loss
  9. Experiments
  10. Datasets and Metrics
  11. Investigated LT Algorithms
  12. Comparisons with LT Line-up
  13. Ablation Studies and Further Analyses
  14. Conclusion
  15. Acknowledgements:
  16. ...and 12 more sections

Figures (10)

  • Figure 1: (a) The real-world long-tailed distribution is both class-wise and attribute-wise imbalanced; (b) even if we balance the class distribution of MSCOCO-Attribute patterson2016coco, the attributes are still long-tailed
  • Figure 2: In ImageNet-GLT, a typical LT method, (b) LWS kang2019decoupling, is playing a precision-accuracy trade-off with the (a) biased cross-entropy baseline, while the proposed (c) IFL improves both metrics at the same time. We follow liu2019largekang2019decoupling to stratify classes into Many, Medium, and Few by the class frequency. The t-SNE hinton2002stochastic of image features further illustrates that IFL features reduce the confusing region between "tractor" and "harvester" caused by the shared attribute "field"
  • Figure 3: Examples of how to balance the attribute distribution for the Test-GBL evaluation environment in the proposed two benchmarks
  • Figure 4: The proposed IFL that first applies different sampling strategies according to the prediction confidence within each class, then combines them to construct environments with diverse attribute distributions, and finally learns features invariant to the environment change by the IFL metric loss
  • Figure 5: (a-b) The trending of precision and accuracy after applying the IFL; (c-d) GLT baselines will automatically improve class-wise LT, while conventional LT re-balancing algorithms won't improve the attribute-wise imbalance in GLT
  • ...and 5 more figures