PIG: Prompt Images Guidance for Night-Time Scene Parsing

TL;DR

Prompt Images Guidance (PIG) addresses the scarcity of labeled night-time data by coupling a Night-Focused Network (NFNet) with unsupervised domain adaptation (UDA) and a pseudo-label fusion mechanism guided by domain similarity using LPIPS. The method introduces two data augmentations, Prompt Mixture Strategy (PMS) and Alternate Mask Strategy (AMS), to learn from a small set of night-time prompts while avoiding overfitting. PDSG (pseudo-label fusion via domain similarity guidance) blends predictions from UDA and NFNet to produce high-quality pseudo-labels that supervise both networks, yielding substantial gains on NightCity, NightCity+, Dark Zurich, and ACDC without requiring paired day-night data. Empirical results demonstrate that PIG consistently improves strong UDA baselines, validating the efficacy of class-wise fusion and prompt-based night knowledge for robust cross-domain night-time scene understanding.

Abstract

Night-time scene parsing aims to extract pixel-level semantic information in night images, aiding downstream tasks in understanding scene object distribution. Due to limited labeled night image datasets, unsupervised domain adaptation (UDA) has become the predominant method for studying night scenes. UDA typically relies on paired day-night image pairs to guide adaptation, but this approach hampers dataset construction and restricts generalization across night scenes in different datasets. Moreover, UDA, focusing on network architecture and training strategies, faces difficulties in handling classes with few domain similarities. In this paper, we leverage Prompt Images Guidance (PIG) to enhance UDA with supplementary night knowledge. We propose a Night-Focused Network (NFNet) to learn night-specific features from both target domain images and prompt images. To generate high-quality pseudo-labels, we propose Pseudo-label Fusion via Domain Similarity Guidance (FDSG). Classes with fewer domain similarities are predicted by NFNet, which excels in parsing night features, while classes with more domain similarities are predicted by UDA, which has rich labeled semantics. Additionally, we propose two data augmentation strategies: the Prompt Mixture Strategy (PMS) and the Alternate Mask Strategy (AMS), aimed at mitigating the overfitting of the NFNet to a few prompt images. We conduct extensive experiments on four night-time datasets: NightCity, NightCity+, Dark Zurich, and ACDC. The results indicate that utilizing PIG can enhance the parsing accuracy of UDA.

Figures (10)

• Figure 1: In unsupervised domain adaptation, the adaptation result tends to worsen as the similarity of object classes between the source and target domains decreases.
• Figure 2: The training pipeline of the Prompt Images Guidance (PIG). The datasets include labeled day images from the source domain, a labeled set of prompt images, and unlabeled night images from the target domain. UDA trains on both day and night images. When processing target domain images, a Cross-Domain Mixed (CDM) data augmentation method is commonly employed for both night and day images. NFNet exclusively trains on night images, where the input night images are combined with the prompt images using the Prompt Mixture Strategy (PMS) and the Alternate Mask Strategy (AMS). The predictions $\hat{y}^{T}_{uda}$ and $\hat{y}^{T}_{nf}$, from both UDA and NFNet are then fed into the FDSG module. FDSG is detailed in Fig. \ref{['fig:FDGG']}. Depending on the ordering of the FDSG results, $\hat{y}^{T}_{uda}$ and $\hat{y}^{T}_{nf}$ are fused with different class weights to produce the pseudo-label $\hat{y}^{T}_{fuse}$. Finally, $\hat{y}^{T}_{fuse}$ is utilized in UDA to calculate UDA losses $L_{UDA}$ with the ground truth from the source domain, while in NFNet, $\hat{y}^{T}_{fuse}$ participates in the augmentation losses $L_{A}$ and prompt losses $L_{P}$ along with the ground truth from the prompt images.
• Figure 3: Pseudo-label Fusion via Domain Similarity Guidance (FDSG). The input day image generates a single-class image that contains only one class, based on the corresponding ground truth. These single-class images are sequentially fed into the LPIPS module alongside the target domain night image. The network evaluates single-class images and night image, producing an evaluation result for each single-class image. A higher evaluation result indicates a greater difficulty in adapting the class during this iteration. We select the first $k$ classes to guide the prediction $\hat{y}^{T}_{nf}$ of NFNet, generating a mask that contains $k$ classes. Additionally, we employ UDA to predict small objects in $\hat{y}^{T}_{uda}$, generating a small object mask. The two masks are then subtracted from each other to obtain the fused mask $M_{fuse}$. Finally, fusion pseudo-label $\hat{y}^{T}_{fuse}$ is generated by combining $\hat{y}^{T}_{nf}$, $\hat{y}^{T}_{uda}$, and $M_{fuse}$ according to formula \ref{['eq_fuse']}.
• Figure 4: T-SNE visualization of datasets for virtual city, day-time, and night-time domains.
• Figure 5: Qualitative comparison of PIG with DAFormer and HRDA on Cityscapes to Dark Zurich. In classes with a large domain gap between day and night such as sky, building, and vegetation, PIG achieved notable improvements.