Completely Weakly Supervised Class-Incremental Learning for Semantic Segmentation

TL;DR

The paper tackles completely weakly supervised class-incremental semantic segmentation (CI-WSSS), learning base and novel classes using only image-level labels. It introduces an uncertainty-guided fusion of pseudo-labels from a localizer and a sequence of foundation models, and an exemplar-guided data augmentation strategy to mitigate forgetting. The CI-WSSS framework is validated on 15-5 VOC, 10-10 VOC, and COCO-to-VOC in disjoint and overlap settings, showing superior performance to prior partially weakly supervised methods for novel classes and competitive results overall. This approach reduces annotation costs while maintaining strong segmentation performance, offering practical value for real-world incremental learning with weak supervision.

Abstract

This work addresses the task of completely weakly supervised class-incremental learning for semantic segmentation to learn segmentation for both base and additional novel classes using only image-level labels. While class-incremental semantic segmentation (CISS) is crucial for handling diverse and newly emerging objects in the real world, traditional CISS methods require expensive pixel-level annotations for training. To overcome this limitation, partially weakly-supervised approaches have recently been proposed. However, to the best of our knowledge, this is the first work to introduce a completely weakly-supervised method for CISS. To achieve this, we propose to generate robust pseudo-labels by combining pseudo-labels from a localizer and a sequence of foundation models based on their uncertainty. Moreover, to mitigate catastrophic forgetting, we introduce an exemplar-guided data augmentation method that generates diverse images containing both previous and novel classes with guidance. Finally, we conduct experiments in three common experimental settings: 15-5 VOC, 10-10 VOC, and COCO-to-VOC, and in two scenarios: disjoint and overlap. The experimental results demonstrate that our completely weakly supervised method outperforms even partially weakly supervised methods in the 15-5 VOC and 10-10 VOC settings while achieving competitive accuracy in the COCO-to-VOC setting.

Figures (7)

• Figure 1: Overview of the proposed framework for completely weakly-supervised class-incremental learning for semantic segmentation.
• Figure 2: Proposed framework during training. Pink and green boxes denote inputs including the exemplar set ($\mathcal{E}$) and outputs from the network, respectively. Yellow and blue boxes represent the modules in the network and the losses used for training, respectively.
• Figure 3: Soft pseudo-label generation. $\boldsymbol{M}^{soft\text{-}psd}$, $\boldsymbol{W}^{psd}$, $\boldsymbol{M}^{fdt}$, and $\boldsymbol{M}^{loc}$ denote the final soft pseudo-labels, entropy-based weighting coefficients, pseudo-labels from the foundation models, and those from the localizer $f_l$, respectively.
• Figure 4: Exemplar-guided data augmentation. $\boldsymbol{I}$ and $\mathcal{E}^{t-1}$ represent a training image from the current task and the exemplar set from the previous task, respectively.
• Figure 5: Results of exemplar-guided data augmentation. (Left) Current image; (Middle) Object image from an exemplar set; (Right) Result of augmentation.