Towards Source-free Domain Adaptive Semantic Segmentation via Importance-aware and Prototype-contrast Learning

TL;DR

The paper tackles SFDA for semantic segmentation in autonomous driving, where source data is unavailable and target labels are absent. It introduces the IAPC framework, combining an importance-aware mechanism (EDIK) to extract domain-invariant knowledge from a pre-trained source model with a prototype-contrast strategy (LDSK) to learn target-domain specifics via a memory-based prototype system. The approach uses losses L_IA, L_IM, L_PS, and L_PE to guide knowledge transfer and prototype-driven adaptation, respectively, achieved end-to-end. Empirical results on GTA5$\rightarrow$Cityscapes and SYNTHIA$\rightarrow$Cityscapes show state-of-the-art SFDA performance, with ablations confirming the effectiveness of both IA and prototype-contrast components and demonstrating robustness across hyperparameters and tasks.

Abstract

Domain adaptive semantic segmentation enables robust pixel-wise understanding in real-world driving scenes. Source-free domain adaptation, as a more practical technique, addresses the concerns of data privacy and storage limitations in typical unsupervised domain adaptation methods, making it especially relevant in the context of intelligent vehicles. It utilizes a well-trained source model and unlabeled target data to achieve adaptation in the target domain. However, in the absence of source data and target labels, current solutions cannot sufficiently reduce the impact of domain shift and fully leverage the information from the target data. In this paper, we propose an end-to-end source-free domain adaptation semantic segmentation method via Importance-Aware and Prototype-Contrast (IAPC) learning. The proposed IAPC framework effectively extracts domain-invariant knowledge from the well-trained source model and learns domain-specific knowledge from the unlabeled target domain. Specifically, considering the problem of domain shift in the prediction of the target domain by the source model, we put forward an importance-aware mechanism for the biased target prediction probability distribution to extract domain-invariant knowledge from the source model. We further introduce a prototype-contrast strategy, which includes a prototype-symmetric cross-entropy loss and a prototype-enhanced cross-entropy loss, to learn target intra-domain knowledge without relying on labels. A comprehensive variety of experiments on two domain adaptive semantic segmentation benchmarks demonstrates that the proposed end-to-end IAPC solution outperforms existing state-of-the-art methods. The source code is publicly available at https://github.com/yihong-97/Source-free-IAPC.

Figures (8)

• Figure 1: The trained source model shows poor performance when directly making predictions on the target images. With the proposed IAPC framework, a significant improvement is achieved in road-driving scene segmentation.
• Figure 2: Task of IAPC. The importance-aware mechanism is employed to extract domain-invariant information $\mathcal{K}_{invar}$. Furthermore, the prototype-contrast strategy is utilized to learn target domain-specific knowledge $\mathcal{K}_{sp}^t$. As a result, the trained segmentation model that performs excellently on the target data is obtained.
• Figure 3: The framework of our proposed IAPC. It effectively addresses the problem of model adaptation when the source domain is inaccessible and the target domain is unlabeled. The augmented image $x^*_t$ is fed into the target model $S_t$, whereas the original image $x_t$ is separately fed into the source model $S_s$ and the memory model $S_m$. Guided by the importance map $w$, the predictions $\hat{p}_t, \hat{y}_t$ from the source model and target predictions $p_t, y$ are combined to extract domain-invariant knowledge under the constraint of $\mathcal{L}_{IA}$. The features $f_m$ from the memory network are used to estimate prototypes, and combined with target features $f_t$ are utilized together by using the prototype-contrast strategy to enable domain-specific knowledge learning within the target domain under the constraint of $\mathcal{L}_{PS}$ and $\mathcal{L}_{PE}$.
• Figure 4: We conducted the pixel-level random sampling of both correct and incorrect predictions by the non-adapted source model on the target data. For each average probability distribution of the selected pixels, we visualized the top two probability values.
• Figure 5: Qualitative results of semantic segmentation adaptation on GTA5 $\rightarrow$ Cityscapes. (From top to bottom: Image, Ground Truth, Source Only, HCL Huang_2021_ModelAdaptation, LD You_2021_DomainAdaptive, IAPC-EDIK, IAPC-LDSK, and IAPC results (best viewed in color).