Task Consistent Prototype Learning for Incremental Few-shot Semantic Segmentation

TL;DR

This study introduces a meta-learning-based prototype approach that encourages the model to learn how to adapt quickly while preserving previous knowledge, and introduces prototype space redistribution learning, which dynamically updates class prototypes to establish optimal inter-prototype boundaries within the prototype space.

Abstract

Incremental Few-Shot Semantic Segmentation (iFSS) tackles a task that requires a model to continually expand its segmentation capability on novel classes using only a few annotated examples. Typical incremental approaches encounter a challenge that the objective of the base training phase (fitting base classes with sufficient instances) does not align with the incremental learning phase (rapidly adapting to new classes with less forgetting). This disconnect can result in suboptimal performance in the incremental setting. This study introduces a meta-learning-based prototype approach that encourages the model to learn how to adapt quickly while preserving previous knowledge. Concretely, we mimic the incremental evaluation protocol during the base training session by sampling a sequence of pseudo-incremental tasks. Each task in the simulated sequence is trained using a meta-objective to enable rapid adaptation without forgetting. To enhance discrimination among class prototypes, we introduce prototype space redistribution learning, which dynamically updates class prototypes to establish optimal inter-prototype boundaries within the prototype space. Extensive experiments on iFSS datasets built upon PASCAL and COCO benchmarks show the advanced performance of the proposed approach, offering valuable insights for addressing iFSS challenges.

Figures (5)

• Figure 1: Illustration of the evaluation protocol and our meta-training process. During the online incremental learning stage, the model undergoes training solely on new classes within each incremental session, while evaluation is conducted on all classes encountered thus far. Our strategy aims to replicate this evaluation protocol during the offline base class training stage. This is accomplished by randomly sampling a large portion of base class images to constitute the pseudo base dataset, with the remaining classes forming the pseudo novel classes. Initially, the model trains on the pseudo base dataset and subsequently adapts to the pseudo novel classes. This approach enables the model to learn how to swiftly identify new classes while retaining the ability to segment previously encountered ones
• Figure 2: The proposed prototype-based approach utilizes masked average pooling (MAP) to derive the novel class prototype. Subsequently, all prototypes are projected into a latent prototype space for redistribution. The resulting prototypes form a new classifier $\mathcal{P}^t$ capable of identifying both base and novel classes. This process is considered as a sequential task of the meta-learning optimization. In the online incremental sessions, the feature extractor remains frozen, and only the prototype projector and segmentation head are updated
• Figure 3: The meta-learning optimization strategy samples pseudo-sequential learning tasks on the base set to perform task training. The meta update process encourages the model to learn in a manner that preserves performance on old classes while effectively adapting to novel classes.
• Figure 4: Visualization of multi-step results under shot setting on the PASCAL dataset.
• Figure 5: Ablation study on coefficient $\lambda$.HM performance in the Single step experiment under 1-shot setting.