Depth-Sensitive Soft Suppression with RGB-D Inter-Modal Stylization Flow for Domain Generalization Semantic Segmentation

TL;DR

This paper tackles domain generalization for semantic segmentation with RGB-D data by addressing depth-map noise and domain gaps. It introduces a novelty DSSS framework comprising an RGB-D inter-modal stylization flow, class-wise soft spatial sensitivity suppression, and an RGB-D soft alignment loss to learn depth-based domain-invariant features while preserving complementary RGB information. The approach achieves state-of-the-art generalization across five diverse datasets, including challenging low-light scenarios, and outperforms both RGB-only and RGB-D DG methods while maintaining competitive computational costs. These contributions advance robust multi-class DGSS by effectively integrating depth cues with RGB in a soft, spatially-aware manner.

Abstract

Unsupervised Domain Adaptation (UDA) aims to align source and target domain distributions to close the domain gap, but still struggles with obtaining the target data. Fortunately, Domain Generalization (DG) excels without the need for any target data. Recent works expose that depth maps contribute to improved generalized performance in the UDA tasks, but they ignore the noise and holes in depth maps due to device and environmental factors, failing to sufficiently and effectively learn domain-invariant representation. Although high-sensitivity region suppression has shown promising results in learning domain-invariant features, existing methods cannot be directly applicable to depth maps due to their unique characteristics. Hence, we propose a novel framework, namely Depth-Sensitive Soft Suppression with RGB-D inter-modal stylization flow (DSSS), focusing on learning domain-invariant features from depth maps for the DG semantic segmentation. Specifically, we propose the RGB-D inter-modal stylization flow to generate stylized depth maps for sensitivity detection, cleverly utilizing RGB information as the stylization source. Then, a class-wise soft spatial sensitivity suppression is designed to identify and emphasize non-sensitive depth features that contain more domain-invariant information. Furthermore, an RGB-D soft alignment loss is proposed to ensure that the stylized depth maps only align part of the RGB features while still retaining the unique depth information. To our best knowledge, our DSSS framework is the first work to integrate RGB and Depth information in the multi-class DG semantic segmentation task. Extensive experiments over multiple backbone networks show that our framework achieves remarkable performance improvement.

Figures (9)

• Figure 1: Comparative visualization of RGB and depth images and their edge detection results for the synthetic SYNTHIA datasetros2016synthia (image pair 1) and the real-world InfraParis datasetfranchi2024infraparis (image pair 2). The depth images are colorized for easier visual comparison, while the original depth images are grayscale single-channel images.
• Figure 2: Flowchart of the proposed framework including the RGB-D inter-modal stylization flow, the class-wise soft spatial sensitivity suppression, and the RGB-D soft alignment loss. Only in the training phase are the proposed components applied to facilitate the generation of domain-invariant representations.
• Figure 3: The illustration of the proposed RGB-D inter-modal stylization flow. The size of random cropping is typically set to 64$\times$64, and the random value $\lambda$ can only be between 0 and 1.
• Figure 4: The illustration of the proposed class-wise soft spatial sensitivity suppression. $Z^{d-style}$, $Y_{x}$ and $Z^{d}$ are consistent with those mentioned above.
• Figure 5: Visualization comparison between our DSSS approach and state-of-the-art methods for the GTA5-to-other-datasets generalization task using heavyweight backbones.