Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AIR: Analytic Imbalance Rectifier for Continual Learning

Di Fang, Yinan Zhu, Runze Fang, Cen Chen, Ziqian Zeng, Huiping Zhuang

TL;DR

AIR tackles data-imbalance in continual learning by introducing Analytic Imbalance Rectifier (AIR) with an Analytic Re-weighting Module (ARM) that assigns per-class weights $\\pi_y$ to balance each class’s loss contribution. The classifier remains a closed-form ridge-regression solution, $\\bar{W}_k = (\\sum_y \\pi_y A_{1:k}^{(y)} + \\gamma I)^{-1} (\\sum_y \\pi_y C_{1:k}^{(y)})$, and AIR updates these aggregates online; Generalized AIR extends this approach to GCIL by storing per-class summaries. Empirical results on LT-CIL and GCIL benchmarks show AIR surpasses or matches state-of-the-art methods, especially in imbalanced and privacy-constrained settings, while maintaining non-forgetting behavior. The work provides a practical, exemplar-free, mathematically grounded path toward robust real-world continual learning with imbalanced data.

Abstract

Continual learning enables AI models to learn new data sequentially without retraining in real-world scenarios. Most existing methods assume the training data are balanced, aiming to reduce the catastrophic forgetting problem that models tend to forget previously generated data. However, data imbalance and the mixture of new and old data in real-world scenarios lead the model to ignore categories with fewer training samples. To solve this problem, we propose an analytic imbalance rectifier algorithm (AIR), a novel online exemplar-free continual learning method with an analytic (i.e., closed-form) solution for data-imbalanced class-incremental learning (CIL) and generalized CIL scenarios in real-world continual learning. AIR introduces an analytic re-weighting module (ARM) that calculates a re-weighting factor for each class for the loss function to balance the contribution of each category to the overall loss and solve the problem of imbalanced training data. AIR uses the least squares technique to give a non-discriminatory optimal classifier and its iterative update method in continual learning. Experimental results on multiple datasets show that AIR significantly outperforms existing methods in long-tailed and generalized CIL scenarios. The source code is available at https://github.com/fang-d/AIR.

AIR: Analytic Imbalance Rectifier for Continual Learning

TL;DR

AIR tackles data-imbalance in continual learning by introducing Analytic Imbalance Rectifier (AIR) with an Analytic Re-weighting Module (ARM) that assigns per-class weights to balance each class’s loss contribution. The classifier remains a closed-form ridge-regression solution, , and AIR updates these aggregates online; Generalized AIR extends this approach to GCIL by storing per-class summaries. Empirical results on LT-CIL and GCIL benchmarks show AIR surpasses or matches state-of-the-art methods, especially in imbalanced and privacy-constrained settings, while maintaining non-forgetting behavior. The work provides a practical, exemplar-free, mathematically grounded path toward robust real-world continual learning with imbalanced data.

Abstract

Continual learning enables AI models to learn new data sequentially without retraining in real-world scenarios. Most existing methods assume the training data are balanced, aiming to reduce the catastrophic forgetting problem that models tend to forget previously generated data. However, data imbalance and the mixture of new and old data in real-world scenarios lead the model to ignore categories with fewer training samples. To solve this problem, we propose an analytic imbalance rectifier algorithm (AIR), a novel online exemplar-free continual learning method with an analytic (i.e., closed-form) solution for data-imbalanced class-incremental learning (CIL) and generalized CIL scenarios in real-world continual learning. AIR introduces an analytic re-weighting module (ARM) that calculates a re-weighting factor for each class for the loss function to balance the contribution of each category to the overall loss and solve the problem of imbalanced training data. AIR uses the least squares technique to give a non-discriminatory optimal classifier and its iterative update method in continual learning. Experimental results on multiple datasets show that AIR significantly outperforms existing methods in long-tailed and generalized CIL scenarios. The source code is available at https://github.com/fang-d/AIR.
Paper Structure (34 sections, 1 theorem, 11 equations, 7 figures, 2 tables, 2 algorithms)

This paper contains 34 sections, 1 theorem, 11 equations, 7 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Conventional CIL
  4. Classic CL Techniques
  5. Analytic Continual Learning (ACL)
  6. CIL with Large Pre-trained Models
  7. Long-Tailed CIL (LT-CIL)
  8. Generalized CIL (GCIL)
  9. Method
  10. Class-Incremental Learning Problem
  11. Analytic Classifier for Balanced Dataset
  12. Diagnosis: Classifier Need to Be Rectified
  13. Analytic Imbalance Rectifier (AIR)
  14. Generalized AIR
  15. Experiments
  16. ...and 19 more sections

Key Result

Theorem 1

The global optimal weight of the weighted classifier at phase $k$ is where can be obtained iteratively.

Figures (7)

  • Figure 1: Different settings of imbalanced CIL.
  • Figure 2: The flowchart of AIR, including (a) the input data stream that arrives phase by phase, where data is imbalanced, and the number of classes may change dynamically; (b) a frozen backbone network followed by a buffer layer that extracts features and maps into a higher dimensional space; (c) the analytic re-weighting module (ARM) calculating the re-weighting factor $\pi_{y}$ for each class $\pi_y$; (d) the unbiased classifiers that are iteratively updated at each phase; (d) the frozen backbone network, the frozen buffer layer, and the unbiased classifier are used for inference.
  • Figure 3: Last-phase performance on the testing set of CIFAR-100 under the descending LT-CIL scenario.
  • Figure 4: Last-phase accuracy for classes in each phase.
  • Figure 5: L2 norm of the weight for each class in the last-phase classifier under the descending LT-CIL scenario.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Theorem 1
  • proof