Smoothing Matters: Momentum Transformer for Domain Adaptive Semantic Segmentation

TL;DR

<3-5 sentence high-level summary>: The paper addresses the instability of applying local Vision Transformers (ViTs) to domain adaptive semantic segmentation (DA-SS) by identifying a high-frequency learning dynamics problem in target-domain pseudo-labeling and feature alignment. It introduces TransDA, a Momentum Transformer framework that smooths target-domain supervision via a momentum network and uses a dynamic discrepancy mechanism to adaptively weight samples during adversarial alignment. The approach combines a Swin-based local ViT backbone with MoPL and MoFA to stabilize pseudo labels and features, and a domain similarity network to drive weighted adversarial loss. Empirically, TransDA achieves state-of-the-art results on sim2real benchmarks GTA5→Cityscapes and SYNTHIA→Cityscapes, demonstrating robust cross-domain performance and improved adaptation gains over prior methods.

Abstract

After the great success of Vision Transformer variants (ViTs) in computer vision, it has also demonstrated great potential in domain adaptive semantic segmentation. Unfortunately, straightforwardly applying local ViTs in domain adaptive semantic segmentation does not bring in expected improvement. We find that the pitfall of local ViTs is due to the severe high-frequency components generated during both the pseudo-label construction and features alignment for target domains. These high-frequency components make the training of local ViTs very unsmooth and hurt their transferability. In this paper, we introduce a low-pass filtering mechanism, momentum network, to smooth the learning dynamics of target domain features and pseudo labels. Furthermore, we propose a dynamic of discrepancy measurement to align the distributions in the source and target domains via dynamic weights to evaluate the importance of the samples. After tackling the above issues, extensive experiments on sim2real benchmarks show that the proposed method outperforms the state-of-the-art methods. Our codes are available at https://github.com/alpc91/TransDA

Figures (8)

• Figure 1: The conventional training paradigm and change of predictions over training iterations. We sampled images from target domain, Cityscapes cordts2016cityscapes, and plot the L1-distance between the predictions over images on iteration $t$ and next iteration $t+1$ as the lines. The standard deviation of the distances are indicated by the corresponding shadow areas around the lines. See § \ref{['sec:Momentum']} for details. In both (b) and (c) diagrams, higher value means more unsmooth learning dynamics, thus more high-frequency components, since the model always significantly change its prediction iteration by iteration.
• Figure 2: Momentum Transformer Domain Adaptive Semantic Segmentation
• Figure 3: Smooth different supervisions for target domain in self-training and adversarial training. Mo: Momentum Network, PL: for pseudo labels in self-training, FA: for features in adversarial training
• Figure 4: The visualization of feature space, where we map features to 2D space with UMAP mcinnes2018umap. For a clear illustration, we only show two categories, i.e., red for source person, orange for source rider, green for target person, and blue for target rider. Abbreviation: (S)elf (T)raining, (A)dversarial (T)raining. Data Augmentation is the default option and all ST+AT variants employ dynamic discrepancy and momentum network
• Figure 5: Visualizations of domain adaptive semantic segmentation based on Swin ViT at the inference stage. Abbreviation: (S)elf (T)raining, (A)dversarial (T)raining. * denotes employing our proposed smoothing method.

Theorems & Definitions (2)

• Definition 1
• Definition 2