Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Zero-Forgetting CISS via Dual-Phase Cognitive Cascades

Yuquan Lu, Yifu Guo, Zishan Xu, Siyu Zhang, Yu Huo, Siyue Chen, Siyan Wu, Chenghua Zhu, Ruixuan Wang

Abstract

Continual semantic segmentation (CSS) is a cornerstone task in computer vision that enables a large number of downstream applications, but faces the catastrophic forgetting challenge. In conventional class-incremental semantic segmentation (CISS) frameworks using Softmax-based classification heads, catastrophic forgetting originates from Catastrophic forgetting and task affiliation probability. We formulate these problems and provide a theoretical analysis to more deeply understand the limitations in existing CISS methods, particularly Strict Parameter Isolation (SPI). To address these challenges, we follow a dual-phase intuition from human annotators, and introduce Cognitive Cascade Segmentation (CogCaS), a novel dual-phase cascade formulation for CSS tasks in the CISS setting. By decoupling the task into class-existence detection and class-specific segmentation, CogCaS enables more effective continual learning, preserving previously learned knowledge while incorporating new classes. Using two benchmark datasets PASCAL VOC 2012 and ADE20K, we have shown significant improvements in a variety of challenging scenarios, particularly those with long sequence of incremental tasks, when compared to exsiting state-of-the-art methods. Our code will be made publicly available upon paper acceptance.

Zero-Forgetting CISS via Dual-Phase Cognitive Cascades

Abstract

Continual semantic segmentation (CSS) is a cornerstone task in computer vision that enables a large number of downstream applications, but faces the catastrophic forgetting challenge. In conventional class-incremental semantic segmentation (CISS) frameworks using Softmax-based classification heads, catastrophic forgetting originates from Catastrophic forgetting and task affiliation probability. We formulate these problems and provide a theoretical analysis to more deeply understand the limitations in existing CISS methods, particularly Strict Parameter Isolation (SPI). To address these challenges, we follow a dual-phase intuition from human annotators, and introduce Cognitive Cascade Segmentation (CogCaS), a novel dual-phase cascade formulation for CSS tasks in the CISS setting. By decoupling the task into class-existence detection and class-specific segmentation, CogCaS enables more effective continual learning, preserving previously learned knowledge while incorporating new classes. Using two benchmark datasets PASCAL VOC 2012 and ADE20K, we have shown significant improvements in a variety of challenging scenarios, particularly those with long sequence of incremental tasks, when compared to exsiting state-of-the-art methods. Our code will be made publicly available upon paper acceptance.
Paper Structure (32 sections, 3 theorems, 26 equations, 5 figures, 7 tables, 1 algorithm)

This paper contains 32 sections, 3 theorems, 26 equations, 5 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Preliminaries
  5. Strict Parameter Isolation in CISS
  6. Class-Incremental Segmentation via Cognitive Cascades
  7. Experiments
  8. Setup
  9. Main Results
  10. Ablation Studies
  11. Sensitivity Study
  12. Conclusion
  13. Theoretical Proofs
  14. Forgetting Rate Analysis
  15. Other Zero Forgetting strategy
  16. ...and 17 more sections

Key Result

Theorem 3.3

For any continuous learning algorithm that satisfies Assumption assumption:convergence, (1) if $\bar{\mathcal{E}}_{\tau}(\theta_{\tau}^*)=0$, $\forall\tau<t$, then $\bar{\mathcal{E}}_t(\theta_t^*)=\frac{1}{2(t-1)}\Delta_{t}^{\intercal}\left(\sum_{i=1}^{t-1}\mathbf{H}_{i}(\theta_i^*)\right)\Delta_{t}

Figures (5)

  • Figure 1: mIoU of classes learned at each task in PASCAL VOC continual semantic segmentation under 2-2 (left) and 4-2 (right) settings. While existing methods exhibit declining learning capability as tasks progress, our CogCaS consistently achieves strong performance around 70% mloU,demonstrating stable and robust learning ability across all incremental stages.
  • Figure 2: Demonstration of existing typical CISS framework and the proposed CogCaS framework. (A) Traditional CISS framework consists of a set of segmentation heads which are trained sequentially and activated for inference simultaneously. (B) our proposed CogCaS restructures the traditional CISS formulation into a dual-phase cascade using multi-label classifier and class-specific segmentation head. The multi-label classifier determine whether each learned class exists in the image, and only segmentation heads corresponding to existing classes are activated to produce a foreground-background mask. These masks are then fused to obtain the final segmentation mask using a mask fusion strategy.
  • Figure 3: Representative segmentation results from different methods after the model learns all tasks in the Pascal VOC 2012 15-1 setting.
  • Figure 4: The figure illustrate near-ood samples and process.
  • Figure 5: Impact on model performance when the task-shared parameters are set, evaluated on the VOC 2-2 setting.

Theorems & Definitions (6)

  • Definition 3.2: Average Forgetting Rate
  • Theorem 3.3: Zero-forgetting Condition
  • Lemma 1.1: Task Prior Decomposition
  • proof
  • Theorem 1.2: Cascade Factorization
  • proof