Background Adaptation with Residual Modeling for Exemplar-Free Class-Incremental Semantic Segmentation

TL;DR

Class Incremental Semantic Segmentation (CISS) suffers from catastrophic forgetting and shifting backgrounds when learning new classes in an exemplar-free setting. The authors propose a Background Adaptation mechanism that models the background as residual updates via an additional adaptation channel and aggregates these residuals across steps, along with PB-BCE and Background Adaptation losses, plus Group Knowledge Distillation and Background Feature Distillation to safeguard old knowledge. The method is evaluated on Pascal VOC 2012 and ADE20K under overlapping protocols, showing state-of-the-art exemplar-free performance, with clear gains on novel classes and reduced forgetting. The work provides a practical, memory-free approach to robustly extend segmentation to many new categories in real-world data streams.

Abstract

Class Incremental Semantic Segmentation~(CISS), within Incremental Learning for semantic segmentation, targets segmenting new categories while reducing the catastrophic forgetting on the old categories.Besides, background shifting, where the background category changes constantly in each step, is a special challenge for CISS. Current methods with a shared background classifier struggle to keep up with these changes, leading to decreased stability in background predictions and reduced accuracy of segmentation. For this special challenge, we designed a novel background adaptation mechanism, which explicitly models the background residual rather than the background itself in each step, and aggregates these residuals to represent the evolving background. Therefore, the background adaptation mechanism ensures the stability of previous background classifiers, while enabling the model to concentrate on the easy-learned residuals from the additional channel, which enhances background discernment for better prediction of novel categories. To precisely optimize the background adaptation mechanism, we propose Pseudo Background Binary Cross-Entropy loss and Background Adaptation losses, which amplify the adaptation effect. Group Knowledge Distillation and Background Feature Distillation strategies are designed to prevent forgetting old categories. Our approach, evaluated across various incremental scenarios on Pascal VOC 2012 and ADE20K datasets, outperforms prior exemplar-free state-of-the-art methods with mIoU of 3.0% in VOC 10-1 and 2.0% in ADE 100-5, notably enhancing the accuracy of new classes while mitigating catastrophic forgetting. Code is available in https://andyzaq.github.io/barmsite/.

Figures (4)

• Figure 1: 3D Visualization of Background Adaptation results. The background logits for step $t$ combine those from step $t-1$ with the adaptation logits learned in step $t$, preventing disorderly adjustments and focusing on residuals. The current step background logits are processed using the Sigmoid function. Note that colors range from red (small values) to blue (large values).
• Figure 2: Overview of our framework that consists of a Background Adaptation mechanism and proposed losses. (a) The Background Adaptation mechanism uses the negative part of the adaptation channel to rectify the background logits. (b) Two Background Adaptation losses separately optimize the regions w/ and w/o novel categories to enhance the performance of the Background Adaptation.
• Figure 3: The step-wise mIoU comparison of our approach with previous methods under the sub-tasks 10-1 and 15-1 of Pascal VOC 2012.
• Figure 4: Qualitative analysis on 15-1 of Pascal VOC 2012. The Baseline and Ours w/o BgA losses are in the $1^{st}$ row and the $4^{th}$ row of Tab. \ref{['tab:ab_com']}. $Ours$ applies additional Background Adaptation losses, compared to the Ours w/o BgA losses.