Cs2K: Class-specific and Class-shared Knowledge Guidance for Incremental Semantic Segmentation

TL;DR

The paper tackles incremental semantic segmentation (ISS) by addressing forgetting through dual knowledge channels: class-specific knowledge represented by old class prototypes $\eta_c$ and class-shared knowledge embodied in old weights $\Theta^{t-1}$. It introduces Cs$^2$K, a plug-and-play framework combining prototype-guided pseudo labeling using prototype proximity and a prototype-guided class adaptation with augmented prototypes $\Gamma_c$ and $\Pi_c$, together with Fisher-information–based weight-guided selective consolidation to selectively fuse old and new weights $\Theta^t$. Empirical results on Pascal VOC 2012 and ADE20K across multiple splits show substantial improvements over state-of-the-art ISS methods, with strong gains on new classes while preserving old ones. The approach demonstrates that jointly leveraging class-specific prototypes and discriminative weight integration is effective for exemplar-free ISS and offers a practical pathway toward robust continual learning in segmentation.

Abstract

Incremental semantic segmentation endeavors to segment newly encountered classes while maintaining knowledge of old classes. However, existing methods either 1) lack guidance from class-specific knowledge (i.e., old class prototypes), leading to a bias towards new classes, or 2) constrain class-shared knowledge (i.e., old model weights) excessively without discrimination, resulting in a preference for old classes. In this paper, to trade off model performance, we propose the Class-specific and Class-shared Knowledge (Cs2K) guidance for incremental semantic segmentation. Specifically, from the class-specific knowledge aspect, we design a prototype-guided pseudo labeling that exploits feature proximity from prototypes to correct pseudo labels, thereby overcoming catastrophic forgetting. Meanwhile, we develop a prototype-guided class adaptation that aligns class distribution across datasets via learning old augmented prototypes. Moreover, from the class-shared knowledge aspect, we propose a weight-guided selective consolidation to strengthen old memory while maintaining new memory by integrating old and new model weights based on weight importance relative to old classes. Experiments on public datasets demonstrate that our proposed Cs2K significantly improves segmentation performance and is plug-and-play.

Figures (6)

• Figure 1: Illustration of challenges for ISS. (a) The decision boundary between old classes potted plant and train undergoes a dramatic change without the guidance from old class prototypes. GT means ground truth. (b) Other methods integrate old and new model weights without discrimination, leading to the integrated model weights biased towards old model weights (remember dog but not recognize sheep).
• Figure 2: Overview of our Cs$^2$K model. It updates model parameters with the proposed prototype-guided pseudo labeling and prototype-guided class adaptation from the class-specific knowledge aspect. Then, the old and new model weights are selectively integrated via the weight-guided selective consolidation to trade off performance between old and new classes from the class-shared knowledge aspect.
• Figure 3: The visualization comparison between pseudo labelss on Pascal VOC 2012 vocdataset.
• Figure 4: Quantitative comparison at each step with different methods for 15-1, 10-1, and 5-3 class incremental segmentation scenarios on Pascal VOC 2012 vocdataset.
• Figure 5: The visualization comparison from the last step on Pascal VOC 2012 vocdataset.