Adaptive Prototype Replay for Class Incremental Semantic Segmentation

TL;DR

Adapter targets the core limitation of prototype replay in class incremental semantic segmentation by aligning old-class representations with continuously updated models. It introduces ADC to dynamically adjust old-class prototypes, UAC to reduce prediction uncertainty and promote compact prototypes, and CPD to maintain discriminability among similar prototypes. Together, these components enable effective, training-free adaptation of replayed distributions and improve performance on challenging multi-step CISS benchmarks, achieving state-of-the-art results on Pascal VOC and ADE20K. The approach demonstrates strong generalization, including across memory-based and transformer-based settings, with practical implications for scalable continual segmentation.

Abstract

Class incremental semantic segmentation (CISS) aims to segment new classes during continual steps while preventing the forgetting of old knowledge. Existing methods alleviate catastrophic forgetting by replaying distributions of previously learned classes using stored prototypes or features. However, they overlook a critical issue: in CISS, the representation of class knowledge is updated continuously through incremental learning, whereas prototype replay methods maintain fixed prototypes. This mismatch between updated representation and fixed prototypes limits the effectiveness of the prototype replay strategy. To address this issue, we propose the Adaptive prototype replay (Adapter) for CISS in this paper. Adapter comprises an adaptive deviation compen sation (ADC) strategy and an uncertainty-aware constraint (UAC) loss. Specifically, the ADC strategy dynamically updates the stored prototypes based on the estimated representation shift distance to match the updated representation of old class. The UAC loss reduces prediction uncertainty, aggregating discriminative features to aid in generating compact prototypes. Additionally, we introduce a compensation-based prototype similarity discriminative (CPD) loss to ensure adequate differentiation between similar prototypes, thereby enhancing the efficiency of the adaptive prototype replay strategy. Extensive experiments on Pascal VOC and ADE20K datasets demonstrate that Adapter achieves state-of-the-art results and proves effective across various CISS tasks, particularly in challenging multi-step scenarios. The code and model is available at https://github.com/zhu-gl-ux/Adapter.

Figures (5)

• Figure 1: Performance comparison of prior works on the Pascal VOC 2012 challenging multi-step overlapped scenarios, where contains the small classes in the initial step and long-term incremental steps.
• Figure 2: Overview of the proposed Adapter. After training of step $t-1$, old-class prototypes are computed and saved. At the current step $t$, representation deviations towards old-class are estimated and prototypes are updated with the ADC strategy. Old-class feature distributions are replayed with updated prototypes for classifiers $g^t\left(\cdot\right)$. The UAC loss enhances the consistency of the same class by aggregating the representation of each class with uncertainty-aware loss. The CPD loss facilitates discrimination between new-class features and updated old-class prototypes, improving the efficiency of prototype replay.
• Figure 3: Qualitative comparison on Pascal VOC 2012 between Adapter and previous methods.
• Figure 4: Qualitative results for the overlapped 15-1 setting on Pascal VOC 2012. Plant, Sheep, Sofa, Train, and TV are new coming classes in corresponding steps.
• Figure 5: Qualitative results for the overlapped 100-10 setting on ADE20K. Hood, Oven, Traffic light, Sconce, Fan, Plate, and Glass are new coming classes in corresponding steps.