UniMatch V2: Pushing the Limit of Semi-Supervised Semantic Segmentation

TL;DR

This work targets semi-supervised semantic segmentation by addressing the bottleneck of relying on outdated backbones. It re-benchmarks and advances UniMatch by upgrading to DINOv2 encoders and simplifying the training pipeline into a single unified augmentation stream with a Complementary Dropout, augmented by an EMA teacher for stable pseudo labeling. The approach yields state-of-the-art results across Pascal, Cityscapes, ADE20K, and COCO, and demonstrates the practicality of scaling up encoders and extending to remote sensing and classification tasks. The findings underscore the importance of modern pre-trained encoders and broader augmentation spaces for unlocking the potential of unlabeled data in dense prediction problems.

Abstract

Semi-supervised semantic segmentation (SSS) aims at learning rich visual knowledge from cheap unlabeled images to enhance semantic segmentation capability. Among recent works, UniMatch improves its precedents tremendously by amplifying the practice of weak-to-strong consistency regularization. Subsequent works typically follow similar pipelines and propose various delicate designs. Despite the achieved progress, strangely, even in this flourishing era of numerous powerful vision models, almost all SSS works are still sticking to 1) using outdated ResNet encoders with small-scale ImageNet-1K pre-training, and 2) evaluation on simple Pascal and Cityscapes datasets. In this work, we argue that, it is necessary to switch the baseline of SSS from ResNet-based encoders to more capable ViT-based encoders (e.g., DINOv2) that are pre-trained on massive data. A simple update on the encoder (even using 2x fewer parameters) can bring more significant improvement than careful method designs. Built on this competitive baseline, we present our upgraded and simplified UniMatch V2, inheriting the core spirit of weak-to-strong consistency from V1, but requiring less training cost and providing consistently better results. Additionally, witnessing the gradually saturated performance on Pascal and Cityscapes, we appeal that we should focus on more challenging benchmarks with complex taxonomy, such as ADE20K and COCO datasets. Code, models, and logs of all reported values, are available at https://github.com/LiheYoung/UniMatch-V2.

Figures (10)

• Figure 1: Performance of various methods across different pre-trained encoders (upper-left legend). Under the ResNet-101 backbone, previous works struggle to further improve the best results. But with a simple update on the backbone (ResNet-101 $\rightarrow$ DINOv2-S $\rightarrow$ DINOv2-B) while keeping the method unchanged, the UniMatch unimatch performance is boosted significantly.
• Figure 2: Illustration of the evolution from FixMatch (a) to our prior UniMatch V1 (b), and to our current UniMatch V2 (c). FixMatch uses the prediction of a weakly-augmented image to supervise the corresponding strongly-augmented image. Based on FixMatch, UniMatch V1 brings a separate feature-level augmentation (i.e., Dropout) stream and an additional image-level augmentation stream. Our UniMatch V2, simpler and stronger than V1, unifies image-level and feature-level augmentations into a single stream (Section \ref{['sec:unimatch_v2_unify']}) and presents complementary Dropout to craft better dual views (Section \ref{['sec:unimatch_v2_complementary']}).
• Figure 3: Comparison between the loss scales under image-level augmentations (e.g., color jittering, CutMix) and feature-level augmentation (i.e., Dropout). Loss incurred by image augmentations is much larger than that of feature augmentation.
• Figure 4: Other potential designs of the learnable streams. (a) A single unified image-level and feature-level augmentation stream. (b) Two image-level augmentation streams. (c) Two unified image-level and feature-level augmentation streams, using random Dropout rather than our complementary Dropout. See Section \ref{['sec:learnable_streams']} and Table \ref{['tab:ablation_streams']} for details.
• Figure 5: Ablation study on the confidence threshold $\tau$ (0.95 by default) used to select high-quality pseudo labels.