SfMamba: Efficient Source-Free Domain Adaptation via Selective Scan Modeling

TL;DR

SfMamba tackles source-free domain adaptation by enhancing a Visual Mamba backbone with Channel-wise Visual State-Space blocks and a Semantic-Consistent Shuffle strategy. The method enables long-range, channel-wise dependency learning while mitigating error accumulation through background patch perturbations and consistency regularization, all under a unified loss with pseudo-label refinements. Empirical results across Office-Home, VisDA-C, Office, and DomainNet-126 show state-of-the-art performance with favorable parameter and computation budgets, demonstrating both robustness to domain shifts and practical efficiency. The approach yields notable improvements over prior SFDA methods and offers a principled direction for leveraging state-space models in domain transfer tasks.

Abstract

Source-free domain adaptation (SFDA) tackles the critical challenge of adapting source-pretrained models to unlabeled target domains without access to source data, overcoming data privacy and storage limitations in real-world applications. However, existing SFDA approaches struggle with the trade-off between perception field and computational efficiency in domain-invariant feature learning. Recently, Mamba has offered a promising solution through its selective scan mechanism, which enables long-range dependency modeling with linear complexity. However, the Visual Mamba (i.e., VMamba) remains limited in capturing channel-wise frequency characteristics critical for domain alignment and maintaining spatial robustness under significant domain shifts. To address these, we propose a framework called SfMamba to fully explore the stable dependency in source-free model transfer. SfMamba introduces Channel-wise Visual State-Space block that enables channel-sequence scanning for domain-invariant feature extraction. In addition, SfMamba involves a Semantic-Consistent Shuffle strategy that disrupts background patch sequences in 2D selective scan while preserving prediction consistency to mitigate error accumulation. Comprehensive evaluations across multiple benchmarks show that SfMamba achieves consistently stronger performance than existing methods while maintaining favorable parameter efficiency, offering a practical solution for SFDA. Our code is available at https://github.com/chenxi52/SfMamba.

Figures (10)

• Figure 1: (a) Grad-CAM Grad-cam visualizations from SHOT using ResNet-101 versus VMamba-Tiny on VisDA-C Visda-c. While ResNet exhibits a constrained receptive field with central bias, VMamba achieves broader spatial awareness. (b) Conceptual illustration of how domain shift induces error propagation in state-space models. In the Real $\to$ Painting task (DomainNet-126 domainnet126), spurious sequence correlations from the source domain misguide state evolution, causing errors to accumulate in the target domain.
• Figure 2: Overview of SfMamba (left). The backbone adopts the encoder from VMamba vmamba, followed by an $L\times$ Channel-wise Visual State-Space (Ch-VSS) block (detailed on the right) to perform bidirectional channel scanning. During target adaptation, the input $I$ is processed to generate a Grad-CAM activation map Grad-cam, where low-activation patches are identified as background and shuffled ($\tilde{I}$) via the Semantic-Consistent Shuffle (SCS) strategy. Predictions from both original and perturbed sequences are regularized through KL divergence minimization.
• Figure 3: Comparison of channel-wise attention mechanisms (operating along the channel dimension $c$) with EMAN EMAN and DA-Mamba DAMamba, integrated into the Mamba backbone with state space modeling across different input sequence elements.
• Figure 4: Comparison of the four-way 2D selective scan sequences before and after background patch shuffling in SCS strategy.
• Figure 5: Hyperparameter analysis: (a) Accuracy of the source-model-only and adapted target model on the Office-Home A$\xrightarrow{}$C task across different Ch-VSS-$\textbf{L}$ayer$\textbf{n}$ configurations; (b) Impact of background ratio $\gamma$ on target accuracy for A$\xrightarrow{}$C and A$\xrightarrow{}$P tasks in Office-Home dataset.