Universal Incremental Learning: Mitigating Confusion from Inter- and Intra-task Distribution Randomness

TL;DR

This work addresses the mismatch between traditional incremental learning (IL) settings and real-world dynamics by proposing Universal Incremental Learning (UIL), where both the type and scale of task increments are unknown. It introduces MiCo, a simple yet effective framework that combines a multi-objective loss with direction- and magnitude-decoupled gradient recalibration to mitigate inter-task and intra-task distribution randomness, respectively. Through extensive experiments on iDigits, CORe50, and DomainNet, MiCo achieves state-of-the-art performance in UIL and competitive results in VIL, significantly reducing forgetting while maintaining accurate predictions across evolving task distributions. The findings demonstrate that explicit handling of prediction determinism and balanced gradient updates is crucial for robust continual learning in highly dynamic environments.

Abstract

Incremental learning (IL) aims to overcome catastrophic forgetting of previous tasks while learning new ones. Existing IL methods make strong assumptions that the incoming task type will either only increases new classes or domains (i.e. Class IL, Domain IL), or increase by a static scale in a class- and domain-agnostic manner (i.e. Versatile IL (VIL)), which greatly limit their applicability in the unpredictable and dynamic wild. In this work, we investigate $\textbf{Universal Incremental Learning (UIL)}$, where a model neither knows which new classes or domains will increase along sequential tasks, nor the scale of the increments within each task. This uncertainty prevents the model from confidently learning knowledge from all task distributions and symmetrically focusing on the diverse knowledge within each task distribution. Consequently, UIL presents a more general and realistic IL scenario, making the model face confusion arising from inter-task and intra-task distribution randomness. To $\textbf{Mi}$tigate both $\textbf{Co}$nfusion, we propose a simple yet effective framework for UIL, named $\textbf{MiCo}$. At the inter-task distribution level, we employ a multi-objective learning scheme to enforce accurate and deterministic predictions, and its effectiveness is further enhanced by a direction recalibration module that reduces conflicting gradients. Moreover, at the intra-task distribution level, we introduce a magnitude recalibration module to alleviate asymmetrical optimization towards imbalanced class distribution. Extensive experiments on three benchmarks demonstrate the effectiveness of our method, outperforming existing state-of-the-art methods in both the UIL scenario and the VIL scenario. Our code will be available at $\href{https://github.com/rolsheng/UIL}{here}$.

Figures (4)

• Figure 1: Comparison of existing IL scenarios and our UIL scenario on a 6-class dataset with 4 domains. The horizontal axis denotes class index and the vertical axis denotes domain index. Each incremental task is outlined in black. CIL, DIL, and VIL assume that each incoming task introduces only class increments, domain increments, or both class and domain increments, respectively. In contrast, UIL is a more general IL scenario than existing ones by simultaneously introducing randomness in both the type and scale of increments, as the model neither knows which new classes or domains will increase across all task distributions, nor the number of new classes or domains within each task distribution.
• Figure 2: Analysis on entropy distribution in the proposed UIL scenario, as well as the VIL scenario.
• Figure 3: Illustration on the joint relationship among classes, corresponding gradient magnitude and test accuracy in a 3D bar chart. The imbalanced class distribution are resorted in ascending order based on the number of samples. This figure indicates that there is positive trend where higher gradient magnitudes correlate with improved accuracy, offering valuable insights into the performance across different classes.
• Figure 4: Overview of our proposed UIL scenario (left) and our framework MiCo (right). During training on $\mathcal{D}_{train} = \mathcal{D}_t$, MiCo employs a multi-objective learning scheme and introduces direction- and magnitude-decoupled recalibration modules to mitigate confusion from inter- and intra-task distribution randomness. $\mathcal{L}_{ce}$ and $\mathcal{L}_{em}$ jointly enforce the model to make accurate and deterministic predictions with the role of reducing conflict between $\vec{g}_{ce}$ and $\vec{g}_{em}$) via a direction recalibration module. Moreover, a magnitude recalibration module is used to rescale gradient magnitude (e.g., $||g_{ce}||$, $||g_{em}||$), aiming to alleviate asymmetrical optimization towards imbalanced class distribution. Finally, gradient $\hat{g}_{f}$ with recalibrated direction $\vec{g}_{f}$ and magnitude $||\hat{g}_{f}||$ is used to update parameters in backward propagation. During inference, all previous tasks $\mathcal{D}_{test} = \{\mathcal{D}_{1},\cdots,\mathcal{D}_{t}\}$ are evaluated.