Towards Realistic Incremental Scenario in Class Incremental Semantic Segmentation

TL;DR

This work critiques the overlapped data scenario in Class Incremental Semantic Segmentation as impractical for real-world continual learning. It introduces a partitioned scenario that enforces data disjointness while still capturing background shifts, and identifies memory-replay labeling issues observed in prior studies. To address memory-based forgetting under partitioned data, the authors propose MiB-AugM, a simple yet competitive baseline that augments MiB with a memory-focused loss to handle background shifts. Across Pascal VOC experiments, partitioned eliminates overlapping-data biases and MiB-AugM achieves state-of-the-art results on several task configurations, underscoring its practical impact for robust, scalable CISS.

Abstract

This paper addresses the unrealistic aspect of the commonly adopted Continuous Incremental Semantic Segmentation (CISS) scenario, termed overlapped. We point out that overlapped allows the same image to reappear in future tasks with different pixel labels, which is far from practical incremental learning scenarios. Moreover, we identified that this flawed scenario may lead to biased results for two commonly used techniques in CISS, pseudo-labeling and exemplar memory, resulting in unintended advantages or disadvantages for certain techniques. To mitigate this, a practical scenario called partitioned is proposed, in which the dataset is first divided into distinct subsets representing each class, and then the subsets are assigned to each corresponding task. This efficiently addresses the issue above while meeting the requirement of CISS scenario, such as capturing the background shifts. Furthermore, we identify and address the code implementation issues related to retrieving data from the exemplar memory, which was ignored in previous works. Lastly, we introduce a simple yet competitive memory-based baseline, MiB-AugM, that handles background shifts of current tasks in the exemplar memory. This baseline achieves state-of-the-art results across multiple tasks involving learning numerous new classes.

Figures (6)

• Figure 1: (Left) Illustration of data reappearance issue in the overlapped scenario. At task $t$, an image seen at task $t-1$ is given with different labels, which is far from a practical incremental learning scenario. (Right) Illustration of potential problems caused by the reappearance issue in the overlapped scenario.
• Figure 2: (Left) A Venn diagram illustrating the relationship between ground truth class for each datum. Each datum is assigned to a class set based on the object it contains. (Right) Comparison of $\mathcal{D}^{t}$ (highlighted with black line) for each scenario. Table \ref{['tab: appendix dataset config details']} in the Appendix provides a detailed summary of the assigned labels for each image in every incremental task.
• Figure 3: Illustration of dataset construction steps for partitioned scenario.
• Figure 4: Results of each method on 5-3 Task
• Figure 5: An overview of Pseudo-labeling Retrieval Rate (PRR) metric evaluation