Stable Neighbor Denoising for Source-free Domain Adaptive Segmentation

TL;DR

This work addresses SFUDA for semantic segmentation by tackling pseudo-label noise concentrated in unstable samples. It introduces Stable Neighbor Denoising (SND), which combines a stability-based sample split with bi-level optimization that aligns unstable updates to stable neighbors retrieved via domain proxies for style and layout, plus category compensation through object paste. The method is model-agnostic and demonstrates state-of-the-art performance across multiple SFUDA benchmarks, including synthetic-to-real and open-compound domains, while offering robust ablations and compatibility with other approaches. The proposed approach significantly improves cross-domain robustness and provides a practical, plug-in solution for reducing bias in source-free adaptation pipelines.

Abstract

We study source-free unsupervised domain adaptation (SFUDA) for semantic segmentation, which aims to adapt a source-trained model to the target domain without accessing the source data. Many works have been proposed to address this challenging problem, among which uncertainty-based self-training is a predominant approach. However, without comprehensive denoising mechanisms, they still largely fall into biased estimates when dealing with different domains and confirmation bias. In this paper, we observe that pseudo-label noise is mainly contained in unstable samples in which the predictions of most pixels undergo significant variations during self-training. Inspired by this, we propose a novel mechanism to denoise unstable samples with stable ones. Specifically, we introduce the Stable Neighbor Denoising (SND) approach, which effectively discovers highly correlated stable and unstable samples by nearest neighbor retrieval and guides the reliable optimization of unstable samples by bi-level learning. Moreover, we compensate for the stable set by object-level object paste, which can further eliminate the bias caused by less learned classes. Our SND enjoys two advantages. First, SND does not require a specific segmentor structure, endowing its universality. Second, SND simultaneously addresses the issues of class, domain, and confirmation biases during adaptation, ensuring its effectiveness. Extensive experiments show that SND consistently outperforms state-of-the-art methods in various SFUDA semantic segmentation settings. In addition, SND can be easily integrated with other approaches, obtaining further improvements.

Figures (6)

• Figure 1: Comparison of advantages of different uncertain estimation strategies in self-training method for SFUDA.
• Figure 2: The plot of the mIoU scores versus stability for each target sample throughout training. Stability is calculated by the difference between the initial and the current segmentation. It shows a positive correlation between mIoU (pseudo-label quality) and stability. This observation holds during the whole training process. Experiments are from the GTA5 $\rightarrow$ Cityscapes SFUDA task.
• Figure 3: The pipeline of the proposed Stable Neighbor Denoising (SND). It is formed by the student-teacher modelsohn2020fixmatch. In each optimization, SND performs the inner and outer optimizations sequentially. In the inner loop (red line), SND utilizes the style $\mathcal{Q}_{style}$ and the layout $\mathcal{Q}_{layout}$ factors to retrieve stable neighbors for unstable samples and then performs category compensation to reduce category bias. Thereafter, SND executes Eq. \ref{['meta']} using the teacher model to obtain the unbiased uncertainty map $\omega_{\star}$ and initial pseudo-label $\hat{y}_{t}$. In the outer loop (black line), SND performs Eq. \ref{['outer_1']} to optimize the student model. EMA denotes the Exponential Moving Average.
• Figure 4: Loss gradient directions in bi-level optimization. It is shown by computing the average gradient direction of different sets of classifier weights (ASPP) using the true labels. The black lines are inner-loop and the other colored lines are outer-loop.
• Figure 5: Visualization of different uncertainty estimation results on both GTA $\rightarrow$ Cityscapes and GTA $\rightarrow$ BDD100k tasks. Diff.GT denotes the ground truth estimation mask. Entropy map is shown by probability entropyvu2019advent. Prototype map is shown by the difference between the Aspp classifier and prototype classifier zhang2021prototypicalshen2023diga.