MgSvF: Multi-Grained Slow vs. Fast Framework for Few-Shot Class-Incremental Learning

TL;DR

This paper proposes a multi-grained SvF learning strategy to cope with the SvF dilemma from two different grains: intra-space (within the same feature space) and inter-space (between two different feature spaces).

Abstract

As a challenging problem, few-shot class-incremental learning (FSCIL) continually learns a sequence of tasks, confronting the dilemma between slow forgetting of old knowledge and fast adaptation to new knowledge. In this paper, we concentrate on this "slow vs. fast" (SvF) dilemma to determine which knowledge components to be updated in a slow fashion or a fast fashion, and thereby balance old-knowledge preservation and new-knowledge adaptation. We propose a multi-grained SvF learning strategy to cope with the SvF dilemma from two different grains: intra-space (within the same feature space) and inter-space (between two different feature spaces). The proposed strategy designs a novel frequency-aware regularization to boost the intra-space SvF capability, and meanwhile develops a new feature space composition operation to enhance the inter-space SvF learning performance. With the multi-grained SvF learning strategy, our method outperforms the state-of-the-art approaches by a large margin.

Figures (9)

• Figure 1: Analysis of intra-space SvF on CIFAR100. The forgetting of previous tasks is estimated with average forgettingyu2020semanticchaudhry2018riemannian. (a) Results with 2 learning sessions. (b) Results with 10 learning sessions. It can be seen that different frequency components appear different characteristics for old-knowledge transfer. In both learning settings, the lower frequency components achieve less forgetting, and the average forgetting increases along with the frequency.
• Figure 2: Illustration of slow vs. fast analysis for few-shot class-incremental learning. (a) mainly pays attention to slow forgetting. Samples of old tasks are separated by a large margin but that of new tasks are mixed up. (b) puts emphasis on fast adaptation. New-task samples are separable while old-task samples are mixed up. (c) keeps a trade-off between slow-forgetting and fast-adaptation and solves all tasks well.
• Figure 3: Illustration of our method. The dash lines show the frequency-aware intra-space regularization, and the solid lines indicate the inter-space composition operation. At the $1$-st session, a base embedding model is initially trained on a large-scale training set of base task. At the $t$-th ($t \textgreater 1$) learning session, two embedding models are fast or slowly updated on data of $t$-th task by intra-space SvF learning, then we composite the slow-updated feature space and the fast-updated feature space, and finally use the composite feature space for classification.
• Figure 4: (Best viewed in color.) Visualization of samples in "the unified feature space" or "the composite feature space" by t-SNE on CIFAR100. Samples of ten classes are from two tasks and each class is represented by one color. (a): Samples in the unified feature space at an early session and a later session; (b): Samples in the composite feature space at an early session and a later session.
• Figure 5: (a): Comparison results on CIFAR100 with ResNet18 using the $5$-way $5$-shot FSCIL setting. (b): Comparison results on MiniImageNet with ResNet18 using the $5$-way $5$-shot FSCIL setting. Our method shows clear superiority and outperforms all other methods at each encountered learning session. (c): Change of average accuracy when varying $a$ on CUB200.The performance peaks on an intermediate value, which indicates the importance and complementarity of both the slow-updated space and the fast-updated space.