Tendency-driven Mutual Exclusivity for Weakly Supervised Incremental Semantic Segmentation

TL;DR

This work tackles weakly incremental semantic segmentation (WILSS), where new classes must be learned using only image-level labels, creating conflicts between old-model predictions and seed-area-derived new-class masks. The authors propose Teddy, a framework built around a tendency-driven mutual exclusivity (TME) constraint that enforces non-conflicting foreground predictions for old and new classes, aided by the Segment Anything Model (SAM) for robust mask generation. Teddy formulates a TME-constrained bi-level optimization to jointly generate pixel-level pseudo labels for all classes and update model parameters, with a closed-form solution enabling on-the-fly fusion of masks. Extensive experiments on VOC and COCO demonstrate state-of-the-art performance, strong preservation of old knowledge, and robust improvements across single- and multi-step incremental settings, underscoring Teddy’s practical potential for cost-effective, continual semantic segmentation.

Abstract

Weakly Incremental Learning for Semantic Segmentation (WILSS) leverages a pre-trained segmentation model to segment new classes using cost-effective and readily available image-level labels. A prevailing way to solve WILSS is the generation of seed areas for each new class, serving as a form of pixel-level supervision. However, a scenario usually arises where a pixel is concurrently predicted as an old class by the pre-trained segmentation model and a new class by the seed areas. Such a scenario becomes particularly problematic in WILSS, as the lack of pixel-level annotations on new classes makes it intractable to ascertain whether the pixel pertains to the new class or not. To surmount this issue, we propose an innovative, tendency-driven relationship of mutual exclusivity, meticulously tailored to govern the behavior of the seed areas and the predictions generated by the pre-trained segmentation model. This relationship stipulates that predictions for the new and old classes must not conflict whilst prioritizing the preservation of predictions for the old classes, which not only addresses the conflicting prediction issue but also effectively mitigates the inherent challenge of incremental learning - catastrophic forgetting. Furthermore, under the auspices of this tendency-driven mutual exclusivity relationship, we generate pseudo masks for the new classes, allowing for concurrent execution with model parameter updating via the resolution of a bi-level optimization problem. Extensive experiments substantiate the effectiveness of our framework, resulting in the establishment of new benchmarks and paving the way for further research in this field.

Figures (4)

• Figure 1: Illustration of the proposed Teddy framework. A novel tendency-driven relationship of mutual exclusivity is proposed to regulate the interactions between the predictions produced by the seed areas and the previous model. Moreover, by solving a TME constrained bi-level optimization problem, we can generate pixel-level pseudo labels for all classes and update model parameters simultaneously.
• Figure 2: A simple illustration for tendency-driven mutual exclusivity. In each case, the left vector represents $R(f_{\theta^{t-1}}(\mathbf{x}^t),\alpha)_i$, and the right one represents $\delta(S(\mathbf{x}^t, \mathbf{y}^t))_i$. TME will prevent the case under the green shadow while allowing the other three to occur.
• Figure 3: Qualitative results to show the effectiveness of TME. From left to right: image, binarized predictions for the old classes, seed areas without TME for the new classes, seed areas with TME for the new classes and refined seed areas with TME.
• Figure 4: Qualitative results on 15-5 VOC setting for both new (sheep) and old classes. From left to right: image, WILSON cermelli2022incrementalwilson, Teddy and the ground-truth.