Exploring Semantic Consistency and Style Diversity for Domain Generalized Semantic Segmentation

TL;DR

DGSS faces domain shift when target data are unavailable; the proposed SCSD framework exploits semantic consistency via a Semantic Query Booster and category-aware text embeddings, and enhances cross-domain style diversity through a Text-Driven Style Transform guided by domain-difference prompts. It also introduces Style Synergy Optimization to prevent inter-domain feature collapse by jointly optimizing a style-contrastive loss and a style-aggregation loss, with adaptive weighting. The approach leverages CLIP for image-text alignment and applies Fourier-based modulation to low-frequency components for controllable style transfer in training, achieving end-to-end efficiency without external domain data. Empirically, SCSD achieves state-of-the-art generalization on multiple unseen domains (e.g., average $mIoU=49.11$ on four targets when trained on GTAV), with notable gains in multi-source and adverse-condition settings, underscoring practical impact for robust DGSS.

Abstract

Domain Generalized Semantic Segmentation (DGSS) seeks to utilize source domain data exclusively to enhance the generalization of semantic segmentation across unknown target domains. Prevailing studies predominantly concentrate on feature normalization and domain randomization, these approaches exhibit significant limitations. Feature normalization-based methods tend to confuse semantic features in the process of constraining the feature space distribution, resulting in classification misjudgment. Domain randomization-based methods frequently incorporate domain-irrelevant noise due to the uncontrollability of style transformations, resulting in segmentation ambiguity. To address these challenges, we introduce a novel framework, named SCSD for Semantic Consistency prediction and Style Diversity generalization. It comprises three pivotal components: Firstly, a Semantic Query Booster is designed to enhance the semantic awareness and discrimination capabilities of object queries in the mask decoder, enabling cross-domain semantic consistency prediction. Secondly, we develop a Text-Driven Style Transform module that utilizes domain difference text embeddings to controllably guide the style transformation of image features, thereby increasing inter-domain style diversity. Lastly, to prevent the collapse of similar domain feature spaces, we introduce a Style Synergy Optimization mechanism that fortifies the separation of inter-domain features and the aggregation of intra-domain features by synergistically weighting style contrastive loss and style aggregation loss. Extensive experiments demonstrate that the proposed SCSD significantly outperforms existing state-of-theart methods. Notably, SCSD trained on GTAV achieved an average of 49.11 mIoU on the four unseen domain datasets, surpassing the previous state-of-the-art method by +4.08 mIoU. Code is available at https://github.com/nhw649/SCSD.

Figures (7)

• Figure 1: Illustration of our motivations. Semantic Consistency: Similarity map between domain-irrelevant general text embeddings and image features in different domains. Cross-domain consistent prediction can be achieved through general text prompts. Style Diversity: Simplified version of t-SNE visualization of image features. Domain text difference embeddings are used as style difference features guide the style transformation of image features from source domain to target domain.
• Figure 2: Column 1: Image and ground truth (GT). Columns 2-3: Feature normalization-based and domain randomization-based methods. For both methods, Row 1 shows the t-SNE visualization of feature maps, and Row 2 shows the segmentation result. Better view in zoom.
• Figure 3: Overview of our proposed SCSD. The main contribution are: (a) Semantic Query Booster enhances object queries for cross-domain semantic consistency prediction. (b) Text-Driven Style Transform leverages the diversity of the text modality to facilitate the style transformation of image features. (c) Style Synergy Optimization fortifies the separation of inter-domain features and the aggregation of intra-domain features through the collaboration of two loss functions (i.e., $\mathcal{L}_{sc}$ and $\mathcal{L}_{sa}$).
• Figure 4: Column 1: Image and ground truth (GT). Columns 2-3: Prediction masks output by baseline without SQB and our SCSD at the first and sixth layers of the mask decoder.
• Figure 5: t-SNE visualization of image features. Colors denote different categories in (a) and different domains in (b).