PKI: Prior Knowledge-Infused Neural Network for Few-Shot Class-Incremental Learning

TL;DR

The paper tackles few-shot class-incremental learning (FSCIL) by addressing catastrophic forgetting and overfitting through a prior knowledge-infused neural network (PKI). PKI fuses an ensemble of learnable projectors on top of a frozen backbone and an external memory of class means, updating only the new projector and the classifier in each incremental session while leveraging prior projectors. To accommodate resource constraints, two variants PKIV-1 and PKIV-2 reduce projector counts and storage without drastic performance loss. Empirical results on CIFAR100, MiniImageNet, and CUB200 demonstrate state-of-the-art or competitive average accuracy and robust session-wise gains, validating the efficacy of accumulating and reusing prior knowledge across tasks. The work highlights how cascading prior knowledge can stabilize representations and accelerate learning in continual, data-scarce settings.

Abstract

Few-shot class-incremental learning (FSCIL) aims to continually adapt a model on a limited number of new-class examples, facing two well-known challenges: catastrophic forgetting and overfitting to new classes. Existing methods tend to freeze more parts of network components and finetune others with an extra memory during incremental sessions. These methods emphasize preserving prior knowledge to ensure proficiency in recognizing old classes, thereby mitigating catastrophic forgetting. Meanwhile, constraining fewer parameters can help in overcoming overfitting with the assistance of prior knowledge. Following previous methods, we retain more prior knowledge and propose a prior knowledge-infused neural network (PKI) to facilitate FSCIL. PKI consists of a backbone, an ensemble of projectors, a classifier, and an extra memory. In each incremental session, we build a new projector and add it to the ensemble. Subsequently, we finetune the new projector and the classifier jointly with other frozen network components, ensuring the rich prior knowledge is utilized effectively. By cascading projectors, PKI integrates prior knowledge accumulated from previous sessions and learns new knowledge flexibly, which helps to recognize old classes and efficiently learn new classes. Further, to reduce the resource consumption associated with keeping many projectors, we design two variants of the prior knowledge-infused neural network (PKIV-1 and PKIV-2) to trade off a balance between resource consumption and performance by reducing the number of projectors. Extensive experiments on three popular benchmarks demonstrate that our approach outperforms state-of-the-art methods.

Figures (6)

• Figure 1: The network for FSCIL can be separated into two major components ( e.g., a backbone and a classifier). Existing methods (a-c) tend to freeze more components or parameters and make relatively fewer adjustments in the incremental phase. Inspired by that, our approach (d) ensembles more preceding components to retain prior knowledge, effectively synchronizing old-class memorizing and new-class fitting.
• Figure 2: Our prior knowledge-infused neural network (PKI) is a two-phase approach for FSCIL. After a backbone $\phi_b(\textbf{x};\textbf{w}_b)$, a memory $\mathcal{M}^{(t)}$, a projector ensemble $\{\phi_p^{(0)},...,\phi_p^{(t)}\}$ and a classifier $\phi_c(\textbf{x};\textbf{w}_c)$ are incrementally updated in each session. In the base learning $(t=0)$, our approach trains the whole model on $\mathcal{D}^{(0)}$. In the incremental learning $(t>0)$, our approach freezes the backbone and incorporates the prior knowledge injection to jointly train the projector $\phi_p^{(t)}$ and the classifier $\phi_c^{(t)}$ on $\mathcal{D}^{(t)}$ under the supervision of previously learned $\mathcal{M}^{(t-1)}$ and $\{\phi_p^{(0)},...,\phi_p^{(t-1)}\}$.
• Figure 3: Prior knowledge infusion of PKIV-2.
• Figure 4: Representation visualization with t-SNE maaten2008jmlr, which uses the base session and the last session as an example. We randomly select $50$ examples over $20$ base classes and $5$ incremental classes to show the model effect. Symbols '$\bullet$' and '$\blacktriangle$' represent examples of base classes and incremental classes, respectively. (a), (b) and (c) are visual features on CIFAR100, MiniImageNet, and CUB200, respectively. The top row is visual features in the base session, and the bottom row is visual features in the last incremental session.
• Figure 5: Projector initialization in session $1$ on CIFAR100. (a) and (b) initialize the current projector with the projector from the previous session, and in random, respectively.