Boosting Cross-spectral Unsupervised Domain Adaptation for Thermal Semantic Segmentation

TL;DR

This work addresses labeled-data scarcity in thermal semantic segmentation by transferring knowledge from RGB via a two-way cross-spectral unsupervised domain adaptation framework. It introduces Masked Mutual Learning with uncertainty-based intra- and inter-spectral masks and cross-spectral prototypes to facilitate complementary RGB-thermal knowledge exchange, along with a prototypical self-supervised loss for robust nighttime training. The method shows significant improvements over prior UDA methods and competitive performance with supervised approaches on MFNet and KP datasets, validating the effectiveness of cross-spectral alignment and self-supervision in adverse conditions. By enabling reliable thermal segmentation in day and night, the approach has practical implications for robust autonomous driving systems in low-visibility environments.

Abstract

In autonomous driving, thermal image semantic segmentation has emerged as a critical research area, owing to its ability to provide robust scene understanding under adverse visual conditions. In particular, unsupervised domain adaptation (UDA) for thermal image segmentation can be an efficient solution to address the lack of labeled thermal datasets. Nevertheless, since these methods do not effectively utilize the complementary information between RGB and thermal images, they significantly decrease performance during domain adaptation. In this paper, we present a comprehensive study on cross-spectral UDA for thermal image semantic segmentation. We first propose a novel masked mutual learning strategy that promotes complementary information exchange by selectively transferring results between each spectral model while masking out uncertain regions. Additionally, we introduce a novel prototypical self-supervised loss designed to enhance the performance of the thermal segmentation model in nighttime scenarios. This approach addresses the limitations of RGB pre-trained networks, which cannot effectively transfer knowledge under low illumination due to the inherent constraints of RGB sensors. In experiments, our method achieves higher performance over previous UDA methods and comparable performance to state-of-the-art supervised methods.

Figures (4)

• Figure 1: To handle the labeled data scarcity problem in the thermal domain, we employ a pre-trained model on a large-scale RGB dataset CityScapes to train student networks. (a) Previous SOTA method MS-UDA performs one-way distillation from PL to THR, disregarding the characteristics of each spectral domain. (b) Our approach adopts two-way distillation, which appropriately transfers the complementary knowledge of each spectral domain. (c) To validate the potential of the distillation processes, we evaluate the performance of the teacher (i.e., PL) and each spectral student network. Our final method outperforms the previous SOTA method in both RGB and thermal domains despite leveraging the same pseudo-labels at the training phase. Due to the limitations of the RGB sensor at nighttime, previous SOTA and our method leverage only daytime RGB images for the distillation process (i.e., RGB Day). (PL: pseudo-labels, THR: thermal)
• Figure 2: An overview of our framework. (a) In stage 1, both RGB and thermal networks are trained in the daytime using pseudo-labels $y_{PL}$ generated by HRNet HRNet pre-trained on a large-scale RGB dataset CityScapes. Simultaneously, Masked Mutual Learning (MML) is applied between these student networks, and cross-spectral prototypes $\eta_{RT}$ are gradually updated during training time. (b) In stage 2, the same learning process is performed for daytime as in (a). We impose prototypical self-supervised loss $L_{PSL}$ using our cross-spectral prototypes to address the absence of reliable annotations for nighttime training. We note that gray lines indicate processes performed only during the training phase, while red lines indicate processes in both the training and inference phases. The cos also refers to the cosine similarity function.
• Figure 3: We present a conceptual illustration of masks in Masked Mutual Learning (MML) (a), and the visualization results (b). (a) We generate RGB and THR masks by incorporating intra- and inter-spectral masks calculated from uncertainty maps for each spectral segmentation prediction. These masks exploit the strengths of each domain while mitigating its inherent limitations (e.g., person in RGB and bicycle in THR), providing complementary training signals to each spectral student network. (b) The visualization results of our masks during training.
• Figure 4: Qualitative comparison with MS-UDA MS-UDA on the MF MFNet and KP KP datasets. We visualize the prediction result of the daytime image for the MF dataset (first row) and the nighttime image for the KP dataset (second row). In comparison to MS-UDA, our approach shows robust performance for both daytime and nighttime across all classes.