DDF: A Novel Dual-Domain Image Fusion Strategy for Remote Sensing Image Semantic Segmentation with Unsupervised Domain Adaptation

TL;DR

This work tackles the domain-shift problem in remote sensing semantic segmentation under limited labeled data by introducing a hybrid training framework that blends self-training with generative style transfer. A novel dual-domain image fusion (DDF) module creates intermediate-domain samples by merging original and transferred images, while a pseudo-label regional weighting (PRW) scheme emphasizes hard-to-recognize regions, particularly boundaries. The approach, built on SegFormer, achieves notable improvements on ISPRS Potsdam and Vaihingen benchmarks, including gains in mIoU and F1-scores across cross-domain tasks, and demonstrates robustness through extensive ablations and qualitative visualizations. The proposed method holds practical value for deploying RS segmentation models across diverse geographic regions and sensor configurations by mitigating domain gaps with intermediate-domain information and region-aware pseudo-labeling.

Abstract

Semantic segmentation of remote sensing images is a challenging and hot issue due to the large amount of unlabeled data. Unsupervised domain adaptation (UDA) has proven to be advantageous in incorporating unclassified information from the target domain. However, independently fine-tuning UDA models on the source and target domains has a limited effect on the outcome. This paper proposes a hybrid training strategy as well as a novel dual-domain image fusion strategy that effectively utilizes the original image, transformation image, and intermediate domain information. Moreover, to enhance the precision of pseudo-labels, we present a pseudo-label region-specific weight strategy. The efficacy of our approach is substantiated by extensive benchmark experiments and ablation studies conducted on the ISPRS Vaihingen and Potsdam datasets.

Figures (5)

• Figure 1: The use of UDA for remote sensing images encounters numerous obstacles because of variations in geographical locations, temporal changes, and seasonal inconsistencies. Moreover, the utilization of diverse sensors leads to images that possess distinct spectral characteristics, resolutions, and levels of noise. These inherent dissimilarities among datasets contribute to the challenge of models trained on one dataset being able to generalize effectively to another.
• Figure 2: The structure we propose comprises three primary elements: a self-training model that utilizes GT augmented images, a dual-domain fusion module (DDF), and a strategy (PRW) for assigning weights to pseudo-label regions. The DDF integrates the original image from the source domain with its corresponding transferred image into the student model. The teacher model assigns pseudo-labels to the target domain image, and the student model generates predictions based on these labels. We also incorporate regional adjustments to the weights of the pseudo-labels.
• Figure 3: Comprehensive depiction of the proposed CNN Fusion module.
• Figure 4: Comprehensive depiction of the proposed Efficient Fusion module.
• Figure 5: The qualitative visualization of the cross-domain semantic segmentation from Potsdam IR-R-G to Vaihingen IR-R-G.