Vision-Language Model Purified Semi-Supervised Semantic Segmentation for Remote Sensing Images

TL;DR

This work tackles pseudo-label quality in remote-sensing semi-supervised semantic segmentation by coupling a teacher-student S4 framework with a Vision-Language Model pseudo-label purifying module (VLM-PP). VLM-PP refines or replaces low-confidence or conflicting pseudo-labels using external semantic reasoning from a VLM, enabling reliable supervision and rectification of misclassifications, especially at multi-class boundaries, while maintaining RS-specific training. Empirical results on LoveDA and ISPRS-Potsdam demonstrate state-of-the-art performance and improved boundary accuracy, with clear ablations confirming the value of VLM-PP and an interpretable purification process. The approach, implemented with open-source backbones (DINOv2_small) and Qwen-VL, offers practical benefits for RS tasks and provides a baseline for future exploration of VLM integration in semi-supervised remote sensing, including potential extensions to hyperspectral data.

Abstract

The semi-supervised semantic segmentation (S4) can learn rich visual knowledge from low-cost unlabeled images. However, traditional S4 architectures all face the challenge of low-quality pseudo-labels, especially for the teacher-student framework.We propose a novel SemiEarth model that introduces vision-language models (VLMs) to address the S4 issues for the remote sensing (RS) domain. Specifically, we invent a VLM pseudo-label purifying (VLM-PP) structure to purify the teacher network's pseudo-labels, achieving substantial improvements. Especially in multi-class boundary regions of RS images, the VLM-PP module can significantly improve the quality of pseudo-labels generated by the teacher, thereby correctly guiding the student model's learning. Moreover, since VLM-PP equips VLMs with open-world capabilities and is independent of the S4 architecture, it can correct mispredicted categories in low-confidence pseudo-labels whenever a discrepancy arises between its prediction and the pseudo-label. We conducted extensive experiments on multiple RS datasets, which demonstrate that our SemiEarth achieves SOTA performance. More importantly, unlike previous SOTA RS S4 methods, our model not only achieves excellent performance but also offers good interpretability. The code is released at https://github.com/wangshanwen001/SemiEarth.

Figures (10)

• Figure 1: Structure comparison of S4 frameworks: (a) Self-training semi-supervised models, (b)-(c) Consistency regularization models, (d) Our SemiEarth.
• Figure 2: The overall structure of our VLM-purified RS S4, i.e., SemiEarth, consists of unsupervised learning with unlabeled data and supervised learning with labeled data. The pseudo-labels generated by the teacher are not directly provided to the student but first purified through the VLM-PP. Blue lines indicate labeled data flow, yellow lines indicate unlabeled, red lines indicate VLM-PP flow, blue dashed line indicates gradient descent and backpropagation, and the red dashed lines serve as an indicative purpose.
• Figure 3: The core logic of VLM purifying low-quality pseudo-labels from teachers.
• Figure 4: Samples from the two RS Datasets.
• Figure 5: Qualitative results with different SOTA S4 methods on the ISPRS-Potsdam dataset.