Class Incremental Learning with Task-Specific Batch Normalization and Out-of-Distribution Detection

TL;DR

The innovation of this study lies in the first-time introduction of task-specific BN into CIL and verifying the feasibility of extending TIL methods to CIL through task-ID prediction with state-of-the-art performance on multiple datasets.

Abstract

This study focuses on incremental learning for image classification, exploring how to reduce catastrophic forgetting of all learned knowledge when access to old data is restricted due to memory or privacy constraints. The challenge of incremental learning lies in achieving an optimal balance between plasticity, the ability to learn new knowledge, and stability, the ability to retain old knowledge. Based on whether the task identifier (task-ID) of an image can be obtained during the test stage, incremental learning for image classifcation is divided into two main paradigms, which are task incremental learning (TIL) and class incremental learning (CIL). The TIL paradigm has access to the task-ID, allowing it to use multiple task-specific classification heads selected based on the task-ID. Consequently, in CIL, where the task-ID is unavailable, TIL methods must predict the task-ID to extend their application to the CIL paradigm. Our previous method for TIL adds task-specific batch normalization and classification heads incrementally. This work extends the method by predicting task-ID through an "unknown" class added to each classification head. The head with the lowest "unknown" probability is selected, enabling task-ID prediction and making the method applicable to CIL. The task-specific batch normalization (BN) modules effectively adjust the distribution of output feature maps across different tasks, enhancing the model's plasticity.Moreover, since BN has much fewer parameters compared to convolutional kernels, by only modifying the BN layers as new tasks arrive, the model can effectively manage parameter growth while ensuring stability across tasks. The innovation of this study lies in the first-time introduction of task-specific BN into CIL and verifying the feasibility of extending TIL methods to CIL through task-ID prediction with state-of-the-art performance on multiple datasets.

Figures (4)

• Figure 1: The two main paradigms of incremental learning for image classification: task incremental learning (with task-ID available during testing) and class incremental learning (without task-ID during testing).
• Figure 2: Framework of our proposed method. Sub-figures (a) and (b) depict the training stage. Sub-figure (a) shows the process of gradually adding and training task-specific BN layers and classification heads as new tasks arrive. After each task is completed, the OOD capability alignment stage, as shown in sub-figure (b), is conducted. Sub-figure (c) presents the test stage, which starts by identifying the classification head with the lowest output probability of the “unknown” class for task-ID prediction (TP) and finally performing within-task prediction (WP) on these outputs.
• Figure 3: Performance variation curves of different methods on three datasets under different settings.
• Figure 4: Comparison of parameter increase between the proposed method and the best-performing baseline methods, DynaER and MORE, on the CIFAR100 dataset with 10 tasks.