Scribble Hides Class: Promoting Scribble-Based Weakly-Supervised Semantic Segmentation with Its Class Label

TL;DR

This work tackles the high annotation cost of semantic segmentation by advancing scribble-based weakly-supervised segmentation. It introduces the class-driven scribble promotion (CDSP) framework, which leverages image-level class cues to generate globally informed pseudo-labels and combines them with scribble supervision. A localization rectification module (LoRM) and a distance entropy loss (DEL) address noise and uncertainty in pseudo-labels, yielding robust improvements on ScribbleSup. The approach achieves state-of-the-art performance and demonstrates strong resilience to scribble quality, with practical implications for scalable segmentation in real-world data.

Abstract

Scribble-based weakly-supervised semantic segmentation using sparse scribble supervision is gaining traction as it reduces annotation costs when compared to fully annotated alternatives. Existing methods primarily generate pseudo-labels by diffusing labeled pixels to unlabeled ones with local cues for supervision. However, this diffusion process fails to exploit global semantics and class-specific cues, which are important for semantic segmentation. In this study, we propose a class-driven scribble promotion network, which utilizes both scribble annotations and pseudo-labels informed by image-level classes and global semantics for supervision. Directly adopting pseudo-labels might misguide the segmentation model, thus we design a localization rectification module to correct foreground representations in the feature space. To further combine the advantages of both supervisions, we also introduce a distance entropy loss for uncertainty reduction, which adapts per-pixel confidence weights according to the reliable region determined by the scribble and pseudo-label's boundary. Experiments on the ScribbleSup dataset with different qualities of scribble annotations outperform all the previous methods, demonstrating the superiority and robustness of our method.The code is available at https://github.com/Zxl19990529/Class-driven-Scribble-Promotion-Network.

Figures (7)

• Figure 1: Schematic diagrams of different scribble-based WSSS methods. Existing approaches (a-c) overlooked the class label in scribbles, which provides image-level supervision. "P" represents the model prediction. The red dashed line represents the supervision relationship.
• Figure 2: The overview of our method (CDSP). In the first stage, we train a classification model with the image-level class labels extracted from the scribbles to generate the globally considered pseudo-label. Then we train a semantic segmentation model with the globally considered pseudo-label and the scribble label jointly in the second stage. We propose a localization rectification module (LoRM) and a distance entropy loss to assist the training process.
• Figure 3: Visualization results employing resnet50 backbone and deeplabV2 segmentor. (a) is the original image with scribble label, (b) is the pseudo-label for training, (c) is the prediction trained with $\mathcal{L}_{seg}$, (d) is the prediction trained with $\mathcal{L}_{seg}+\mathcal{L}_{lorm}$. (e) is the ground truth label.
• Figure 4: The illuastration of LoRM.
• Figure 5: Visualization of disance maps with different coefficients for pseudo label boundary (b-d) and scribble (f-h)