Learning Spectral-Decomposed Tokens for Domain Generalized Semantic Segmentation

TL;DR

This paper tackles domain generalized semantic segmentation (DGSS) by leveraging Vision Foundation Models (VFMs) and a novel spectral-decomposed token (SET) framework. By decomposing frozen VFM features into amplitude and phase components via the frequency domain, and applying learnable spectral tokens per branch along with an attention optimization, SET emphasizes style-invariant content extraction while stabilizing amplitude-driven refinements. Empirical results across GTAV, SYNTHIA, and CityScapes demonstrate state-of-the-art gains, with additional evidence of cross-VFM generalization and improved robustness under adverse conditions. The approach offers a lightweight yet effective path to domain generalization in segmentation tasks, with potential applicability to other vision tasks using VFMs.

Abstract

The rapid development of Vision Foundation Model (VFM) brings inherent out-domain generalization for a variety of down-stream tasks. Among them, domain generalized semantic segmentation (DGSS) holds unique challenges as the cross-domain images share common pixel-wise content information but vary greatly in terms of the style. In this paper, we present a novel Spectral-dEcomposed Token (SET) learning framework to advance the frontier. Delving into further than existing fine-tuning token & frozen backbone paradigm, the proposed SET especially focuses on the way learning style-invariant features from these learnable tokens. Particularly, the frozen VFM features are first decomposed into the phase and amplitude components in the frequency space, which mainly contain the information of content and style, respectively, and then separately processed by learnable tokens for task-specific information extraction. After the decomposition, style variation primarily impacts the token-based feature enhancement within the amplitude branch. To address this issue, we further develop an attention optimization method to bridge the gap between style-affected representation and static tokens during inference. Extensive cross-domain experiments show its state-of-the-art performance.

Figures (6)

• Figure 1: Segmentation performance on unseen target domain (in mIoU, %) v.s. trainable parameter number (in million, M). GTAV richter2016playing is used as the source domain and CityScapes cordts2016cityscapes is used as unseen target domain. The proposed SET shows state-of-the-art performance with little trainable parameters.
• Figure 2: A toy example on how different styles of a driving scene (left row) impact the amplitude (second row) and the phase component (third row). As the styles change, the phase component remains stable, while the amplitude adjusts correspondingly. Thus, the amplitude component provides a feasible path to inspect the cross-domain style variation.
• Figure 3: Framework overview of the proposed Spectral-dEcomposed Token (SET) learning scheme. Embedded into each frozen layer of a VFM, the proposed SET consists of three steps, namely spectral decomposition (in Sec. \ref{['sec3.2']}), learning spectral tokens (in Sec. \ref{['sec3.3']}) and attention optimization in amplitude branch (in Sec. \ref{['sec3.4']}). By default, DINOv2 oquab2023dinov2 is used as the frozen VFM, while the proposed SET is versatile to different VFMs.
• Figure 4: T-SNE visualization of the feature space from baseline (left), and the proposed SET (RDWT; right). The model is trained on the CityScapes source domain, and inferred on the rest four unseen target domains. The proposed SET allows the unseen target domain samples to be more uniformly distributed. Better zoom in to view.
• Figure 5: Visual segmentation results on unseen target domains under the C $\rightarrow$ B, M, G, S setting. The proposed SET is compared with ISW choi2021robustnet, SAW peng2022semantic, WildNet lee2022wildnet, SPC huang2023style and Rein wei2023stronger.