Improving Semi-Supervised Semantic Segmentation with Dual-Level Siamese Structure Network

TL;DR

This work tackles semi-supervised semantic segmentation by enhancing unlabeled data exploitation through a Dual-Level Siamese Structure Network (DSSN). DSSN introduces pixel-wise contrastive learning across low-level image space and high-level feature space, combined with a class-aware pseudo-label generation (CPLG) that selects high-confidence, per-class pseudo labels for weak-to-strong supervision. The approach yields state-of-the-art results on PASCAL VOC 2012 and Cityscapes, particularly excelling under limited labeled data and imbalanced class distributions. The combination of dual-level contrastive objectives and per-class pseudo-label selection demonstrates a robust strategy for leveraging unlabeled data in segmentation tasks, with public code available for replication.

Abstract

Semi-supervised semantic segmentation (SSS) is an important task that utilizes both labeled and unlabeled data to reduce expenses on labeling training examples. However, the effectiveness of SSS algorithms is limited by the difficulty of fully exploiting the potential of unlabeled data. To address this, we propose a dual-level Siamese structure network (DSSN) for pixel-wise contrastive learning. By aligning positive pairs with a pixel-wise contrastive loss using strong augmented views in both low-level image space and high-level feature space, the proposed DSSN is designed to maximize the utilization of available unlabeled data. Additionally, we introduce a novel class-aware pseudo-label selection strategy for weak-to-strong supervision, which addresses the limitations of most existing methods that do not perform selection or apply a predefined threshold for all classes. Specifically, our strategy selects the top high-confidence prediction of the weak view for each class to generate pseudo labels that supervise the strong augmented views. This strategy is capable of taking into account the class imbalance and improving the performance of long-tailed classes. Our proposed method achieves state-of-the-art results on two datasets, PASCAL VOC 2012 and Cityscapes, outperforming other SSS algorithms by a significant margin. The source code is available at https://github.com/kunzhan/DSSN.

Figures (7)

• Figure 1: Illustration of the motivation. (a) demonstrates the proposed dual-level contrastive structure for exploiting the maximum potential of unlabelled samples. (b) depicts the structure of the vanilla contrastive learning. (c) compares the threshold selection strategies of the proposed class-aware pseudo-label generation method and the classical approaches of utilizing a threshold for all classes.
• Figure 2: The overview of DSSN . Dual-level contrastive learning and weak-to-strong pseudo supervision.
• Figure 3: The proposed DSSN method effectively utilizes unlabeled images, as demonstrated by its performance on the Cityscapes dataset with a 1/30 data split and ResNet-50. Compared to SOTAs, DSSN outperforms them significantly.
• Figure 4: Comparsion CPLG to the fixed threshold.
• Figure 5: The mIoU of four long-tailed classes.