In defense of the two-stage framework for open-set domain adaptive semantic segmentation

TL;DR

This work tackles Open-Set Domain Adaptation for Semantic Segmentation (OSDA-SS) by shifting from a one-stage to a two-stage framework. The Separating-then-Adapting Training Strategy (SATS) first learns to separate known from unknown classes using a $(K+1)$-class head and novel virtual unknowns, then performs unknown-aware domain adaptation with a self-training regime and hard unknown exploration to balance learning across known and unknown classes. Virtual unknown construction and class-mixup-driven augmentation enable robust unknown separation and more accurate target unknown identification, yielding substantial H-Score gains on GTA5→Cityscapes ($+3.85 ext{pp}$) and SYNTHIA→Cityscapes ($+18.64 ext{pp}$) over prior methods. The approach also demonstrates strong ablations and robustness across parameter choices, indicating its applicability to real-world scenarios with target-private class emergence and domain shifts. Overall, SATS contributes a practical, effective paradigm for OSDA-SS that improves safety and reliability in dynamic perception tasks such as autonomous driving and robotics.

Abstract

Open-Set Domain Adaptation for Semantic Segmentation (OSDA-SS) presents a significant challenge, as it requires both domain adaptation for known classes and the distinction of unknowns. Existing methods attempt to address both tasks within a single unified stage. We question this design, as the annotation imbalance between known and unknown classes often leads to negative transfer of known classes and underfitting for unknowns. To overcome these issues, we propose SATS, a Separating-then-Adapting Training Strategy, which addresses OSDA-SS through two sequential steps: known/unknown separation and unknown-aware domain adaptation. By providing the model with more accurate and well-aligned unknown classes, our method ensures a balanced learning of discriminative features for both known and unknown classes, steering the model toward discovering truly unknown objects. Additionally, we present hard unknown exploration, an innovative data augmentation method that exposes the model to more challenging unknowns, strengthening its ability to capture more comprehensive understanding of target unknowns. We evaluate our method on public OSDA-SS benchmarks. Experimental results demonstrate that our method achieves a substantial advancement, with a +3.85% H-Score improvement for GTA5-to-Cityscapes and +18.64% for SYNTHIA-to-Cityscapes, outperforming previous state-of-the-art methods.

Figures (4)

• Figure 1: Visual comparison of the UDA method (MIC hoyer2023mic), OSDA-SS baselines (head-expansion baseline and BUS choe2024open), and our SATS under the OSDA-SS scenario. White pixels represent unknown classes. The UDA method (b) misclassifies all unknowns as known. Existing one-stage OSDA-SS approaches often relabel low-confidence pseudo-labeled pixels as unknown, leading to known classes being misclassified as unknown (highlighted in green boxes of (c) and (d)). Additionally, because known classes are learned faster, they tend to overshadow unknown classes, leading to underfitting of these unknown classes (emphasized in red boxes of (c) and (d)). Instead, our two-stage method (e) overcomes these issues, yielding more accurate segmentation.
• Figure 2: Illustration of our proposed SATS method, which comprises two sequential stages: known/unknown separation and unknown-aware domain adaptation. Known/unknown separation (Section \ref{['sec:stage1']}) aims to learn an expanded head to accurately identify unknown classes. To this end, "virtual unknowns" are constructed within source samples, providing reliable supervision for these unknown classes. Unknown-aware domain adaptation (Section \ref{['sec:stage2']}) begins with pre-training on both source and target domains, where the source data is further enriched with high-confidence unknowns identified from the first stage. This approach balances the learning of known and unknown classes, allowing the pipeline to further explore "hard unknowns" for improved robustness.
• Figure 3: Visualization results of our method alongside competitive baselines, including the conventional CSDA-SS method MIC hoyer2023mic, its head-expansion version (MIC-Head), and the OSDA-SS method BUS choe2024open, on the GTA5$\rightarrow$Cityscapes benchmark. In these visualizations, white masks indicate unknown classes, and GT represents the ground truth.
• Figure 4: Qualitative comparison of our method between the first and second stages on the SYNTHIA $\rightarrow$ Cityscapes benchmark, alongside competitive baselines, including MIC hoyer2023mic and its head-expansion version (MIC-Head).