On the importance of cross-task features for class-incremental learning

TL;DR

This paper tackles class-incremental learning under restricted memory, where cross-task discrimination is required without access to past data. It introduces two replay baselines to isolate cross-task feature learning and a cumulative forgetting metric showing that forgetting is not the principal issue. The results indicate that while cross-task features contribute, the dominant gains come from memory size and improved knowledge transfer between tasks, especially when data per task are scarce. The work suggests future class-IL algorithms should prioritize enhancing cross-task representations and transfer rather than solely preventing forgetting.

Abstract

In class-incremental learning, an agent with limited resources needs to learn a sequence of classification tasks, forming an ever growing classification problem, with the constraint of not being able to access data from previous tasks. The main difference with task-incremental learning, where a task-ID is available at inference time, is that the learner also needs to perform cross-task discrimination, i.e. distinguish between classes that have not been seen together. Approaches to tackle this problem are numerous and mostly make use of an external memory (buffer) of non-negligible size. In this paper, we ablate the learning of cross-task features and study its influence on the performance of basic replay strategies used for class-IL. We also define a new forgetting measure for class-incremental learning, and see that forgetting is not the principal cause of low performance. Our experimental results show that future algorithms for class-incremental learning should not only prevent forgetting, but also aim to improve the quality of the cross-task features, and the knowledge transfer between tasks. This is especially important when tasks contain limited amount of data.

Figures (10)

• Figure 1: Fictive scenario illustrating the two types of features considered. In this scenario, the color is an intra-task feature, and the shape is a cross-task feature. The intra-task features are insufficient to solve the final 4-class problem.
• Figure 2: Average accuracies on CIFAR-100 splitted in 10 tasks (left) and 20 tasks (right) for $FT^{BAL}_{\text{+}CTF}$ and $FT^{BAL}_{\text{-}CTF}$ using 20 exemplars per class compared to their respective upper-bounds (500 exemplars per class). Mean and standard deviation over 10 runs are reported.
• Figure 3: Final average accuracy obtained by each method and their respective upper bounds on CIFAR-100 splitted in 10 tasks (Left) and 20 tasks (Right). The part in red coined "others" can be obtained with better intra-task features, while the orange part is the additional gain obtained when learning cross-task features
• Figure 4: Average accuracies on Imagenet-Subset splitted in 25 tasks for $FT^{BAL}_{\text{+}CTF}$ and $FT^{BAL}_{\text{-}CTF}$ using 20 exemplars per class compared to their respective upper-bounds with maximum memory. As it is the case on cifar, the gap due to the learning of cross-task features is not predominant, and is partly filled by the use of $FT^{BAL}_{\text{+}CTF}$.
• Figure 5: Cumulative accuracies $b_{k}^t$ on CIFAR100 (10 tasks) for $FT^{BAL}_{\text{+}CTF} (20 mem/cls)$ (Left) and $FT^{BAL}_{\text{+}CTF} (max)$ (Right). Grey dashed lines represent $b_{k}^t$ for varying $t$ and one fixed $k$ per line. The blue dotted line represent $b_{t}^t$ for varying $t$, which is the average accuracy.