Pseudo-Label Quality Decoupling and Correction for Semi-Supervised Instance Segmentation

TL;DR

The paper tackles instability in semi-supervised instance segmentation caused by noisy pseudo-labels. It introduces PL-DC, a three-pronged framework with (i) Decoupled Dual-Threshold Filtering to independently gauge class and mask quality, (ii) Dynamic Instance Category Correction using CLIP to mitigate category confusion, and (iii) Pixel-Level Mask Uncertainty-Aware loss to down-weight uncertain mask pixels, all within a teacher-student Mask2Former setting. Empirical results on COCO and Cityscapes show large mAP gains at low labeling ratios (e.g., $+11.6$ mAP with $1\%$ COCO, $+15.5$ mAP with $5\%$ Cityscapes) and robust performance at higher supervision levels. The method advances SSIS by effectively decoupling pseudo-label components and leveraging LVLMs for category correction, with publicly released code expected.

Abstract

Semi-Supervised Instance Segmentation (SSIS) involves classifying and grouping image pixels into distinct object instances using limited labeled data. This learning paradigm usually faces a significant challenge of unstable performance caused by noisy pseudo-labels of instance categories and pixel masks. We find that the prevalent practice of filtering instance pseudo-labels assessing both class and mask quality with a single score threshold, frequently leads to compromises in the trade-off between the qualities of class and mask labels. In this paper, we introduce a novel Pseudo-Label Quality Decoupling and Correction (PL-DC) framework for SSIS to tackle the above challenges. Firstly, at the instance level, a decoupled dual-threshold filtering mechanism is designed to decouple class and mask quality estimations for instance-level pseudo-labels, thereby independently controlling pixel classifying and grouping qualities. Secondly, at the category level, we introduce a dynamic instance category correction module to dynamically correct the pseudo-labels of instance categories, effectively alleviating category confusion. Lastly, we introduce a pixel-level mask uncertainty-aware mechanism at the pixel level to re-weight the mask loss for different pixels, thereby reducing the impact of noise introduced by pixel-level mask pseudo-labels. Extensive experiments on the COCO and Cityscapes datasets demonstrate that the proposed PL-DC achieves significant performance improvements, setting new state-of-the-art results for SSIS. Notably, our PL-DC shows substantial gains even with minimal labeled data, achieving an improvement of +11.6 mAP with just 1% COCO labeled data and +15.5 mAP with 5% Cityscapes labeled data. The code will be public.

Figures (8)

• Figure 1: The proposed PL-DC outperforms the previous state-of-the-art SSIS method, GuidedDistillation berrada2023guided, across all settings. Moreover, PL-DC achieves significant improvements compared to the fully-supervised Mask2Former.
• Figure 2: The relationship between predicted instance scores and the IoU of predicted versus ground-truth instance masks. (a) Predicted instance scores, derived from the product of mask quality and class quality, do not show a positive correlation with IoU. (b) Decoupled mask quality and (c) class quality independently influence the segmentation and classification outcomes of pseudo-labels.
• Figure 3: Confusion matrix of the model trained on $1\%$ COCO. For clarity, we visualize only the $29$ most confused object categories and $1$ background category.
• Figure 4: Framework of our proposed pseudo-label quality decoupling and correction (PL-DC) for semi-supervised instance segmentation. PL-DC includes two segmentation models, both Mask2Former cheng2021mask2former, with identical configurations, namely Teacher and Student. The Teacher model generates an uncertainty map for Pixel-Level Mask Uncertainty-Aware training, filters pseudo-labels by the Decoupled Dual-Threshold Filtering (DDTF) mechanism, and further corrects category by Dynamic Instance Category Correction (DICC). The Teacher's parameters are gradually updated from the Student model via Exponential Moving Average (EMA). The Student is trained using both ground-truth labels and pseudo-labels (with uncertainty map), denoted as $\mathcal{L}_\mathrm{sup}$ and $\mathcal{L}_\mathrm{unsup}$, respectively.
• Figure 5: Segmentation Analysis. We randomly sampled $1k$ images from the COCO train2017 dataset to analysis the segmentation results. They are categorized into $5$ types: correct segmentation (Cor), poor localization (Loc), confusion with similar objects (Sim), confusion with objects of other categories (Oth), and confusion with the background (BG).