Attention Residual Fusion Network with Contrast for Source-free Domain Adaptation

TL;DR

This work tackles source-free domain adaptation (SFDA) by addressing both domain shift and negative transfer without access to source data. It introduces ARFNet, a backbone augmented with Multilayer Attention Residual Fusion (MARF), Global-Local Attention Contrast (GAC), and Dynamic Centroid Evaluation (DCE), leveraging contrastive self-supervision to improve target discrimination and label reliability. MARF fuses discriminative attention across layers, GAC contrasts local and memory-based global views to sharpen inter- and intra-class boundaries, and DCE provides robust, evolving centroids to guide self-distillation. Across five benchmarks, ARFNet achieves state-of-the-art or competitive SFDA performance, demonstrating strong generality across CNN and transformer backbones and highlighting practical potential for privacy-preserving domain adaptation.

Abstract

Source-free domain adaptation (SFDA) involves training a model on source domain and then applying it to a related target domain without access to the source data and labels during adaptation. The complexity of scene information and lack of the source domain make SFDA a difficult task. Recent studies have shown promising results, but many approaches to domain adaptation concentrate on domain shift and neglect the effects of negative transfer, which may impede enhancements of model performance during adaptation. n this paper, addressing this issue, we propose a novel framework of Attention Residual Fusion Network (ARFNet) based on contrast learning for SFDA to alleviate negative transfer and domain shift during the progress of adaptation, in which attention residual fusion, global-local attention contrast, and dynamic centroid evaluation are exploited. Concretely, the attention mechanism is first exploited to capture the discriminative region of the target object. Then, in each block, attention features are decomposed into spatial-wise and channel-wise attentions to achieve the cross-layer attention residual fusion progressively and self-distillation. During adaptation progress, we contrast global and local representations to improve the perceptual capabilities of different categories, which enables the model to discriminate variations between inner-class and intra-class. Finally, a dynamic centroid evaluation strategy is exploited to evaluate the trustworthy centroids and labels for self-supervised self-distillation, which aims to accurately approximate the center of the source domain and pseudo-labels to mitigate domain shift. To validate the efficacy, we execute comprehensive experiments on five benchmarks of varying scales. Experimental outcomes indicate that our method surpasses other techniques, attaining superior performance across SFDA benchmarks.

Figures (6)

• Figure 1: When samples from the real-world domain of Webcom are trained and adapted to fit the Amazon scenario within the Office31 dataset, two significant issues often arise, i.e., noise disturbance and negative transfer.
• Figure 2: The overall workflow of our proposed framework on the target domain. The components of trainable backbone $\mathcal{B}_{t}$ and frozen head $\mathcal{G}_{t}$, they are initialized by the well-trained source parameters of $\mathcal{B}_{s}$ and $\mathcal{G}_{s}$, respectively. The input is global views $x$ and corresponding local views $x^{+}$. In the backbone, an AEM module is introduced to each block, which can capture the attention of the feature. Then, attention and semantic features are fused to obtain fusion feature representations, which are used for MARF. Latent vectors $z$ are obtained from the $\mathcal{B}_{t}$ and projected to feature representations to execute GAC. In the output layer, the DCE is exploited to evaluate the trustworthy centroids and labels for self-supervised self-distillation learning.
• Figure 3: Sensitivity of hyper-parameter $\tau$ and $\lambda$ in specific tasks (Ar$\rightarrow$Cl and Rw$\rightarrow$Cl) on Office-Home Dataset.
• Figure 4: t-SNE visualization for domain adaptation on Office-Home (Ar $\rightarrow$ Cl), Red denotes the source domains, while black denotes the target domains.
• Figure 5: Attention maps of images about clock, mouse, and aircraft in the DomainNet-126 dataset.