Towards Unsupervised Domain Adaptation via Domain-Transformer

TL;DR

This work tackles unsupervised domain adaptation by introducing the Domain-Transformer (DoT), which employs domain-level cross-attention to establish long-range cross-domain correspondences and learns locality-consistent features without relying on pseudo-labels. The authors theoretically connect the attention mechanism to optimal transport, deriving Wasserstein-distance-based generalization bounds that justify the transferability of the learned features. DoT integrates locality-preserving regularization—both unsupervised and supervised—with a plug-and-play design that can work with CNNs or Vision Transformers, and demonstrates strong empirical performance across ImageCLEF, Office-31, Office-Home, VisDA-2017, and DomainNet. The combination of OT-inspired interpretation, locality regularization, and flexible backbones yields state-of-the-art or competitive results, highlighting the practical impact of aligning cross-domain samples through a principled, learnable barycentric mapping.

Abstract

As a vital problem in pattern analysis and machine intelligence, Unsupervised Domain Adaptation (UDA) attempts to transfer an effective feature learner from a labeled source domain to an unlabeled target domain. Inspired by the success of the Transformer, several advances in UDA are achieved by adopting pure transformers as network architectures, but such a simple application can only capture patch-level information and lacks interpretability. To address these issues, we propose the Domain-Transformer (DoT) with domain-level attention mechanism to capture the long-range correspondence between the cross-domain samples. On the theoretical side, we provide a mathematical understanding of DoT: 1) We connect the domain-level attention with optimal transport theory, which provides interpretability from Wasserstein geometry; 2) From the perspective of learning theory, Wasserstein distance-based generalization bounds are derived, which explains the effectiveness of DoT for knowledge transfer. On the methodological side, DoT integrates the domain-level attention and manifold structure regularization, which characterize the sample-level information and locality consistency for cross-domain cluster structures. Besides, the domain-level attention mechanism can be used as a plug-and-play module, so DoT can be implemented under different neural network architectures. Instead of explicitly modeling the distribution discrepancy at domain-level or class-level, DoT learns transferable features under the guidance of long-range correspondence, so it is free of pseudo-labels and explicit domain discrepancy optimization. Extensive experiment results on several benchmark datasets validate the effectiveness of DoT.

Figures (9)

• Figure 1: To deal with UDA, our DoT learns locality consistency across domains, and then transforms the source domain into the target space to reduce the misclassification rate on the target domain. In the locality consistency part, black arrows imply alignment between class-relevant clusters across domains, orange solid lines imply large weights for intra-class samples, and gray dashed lines imply small and even zero weights for inter-class samples.
• Figure 2: Illustration of our method, which consists of three parts, i.e., the shared backbone network, the domain-transformer (DoT), and the shared classifier. In particular, DoT can be further decomposed into the long-range dependency characterization, the OT-like map, and the locality consistency learning phases. It essentially explores the sample correspondence between domains by the novel attention mechanism at domain-level. With the projection from the source (e.g., $\mathbf{F}^s$) onto the subspace of the target domain, the new feature vectors (e.g., $\hat{\mathbf{F}}^s$) are weighted by the most relevant ones in the target domain, and thus, they are expected to be transferable and adaptive across domains. Note that $\mathbf{Y}^s$ represents the ground-truth label of source samples $\mathbf{X}^s$. Best viewed in color.
• Figure 3: Parameter analysis curves of $\lambda_{s}$ and $\lambda_{t}$ on ImageCLEF.
• Figure 4: Heat-maps of attention matrices on ImageCLEF task I$\rightarrow$C. Darker colors represent larger values. Best viewed in color.
• Figure 5: T-SNE visualization of features generated by Source-Only, DeepJDOT, ATM and DoT on ImageCLEF task I$\rightarrow$C and VisDA-2017 task S$\rightarrow$R, respectively. Here, "o" denotes source domain and "+" denotes target domain. Each color denotes one class. Best viewed in color.

Theorems & Definitions (14)

• Definition 1: True Risk
• Definition 2: Seminorm
• Definition 3: Sobolev Seminorm
• Definition 4: Integral Probability Metric (IPM)
• Remark 1
• Lemma 1
• proof
• Theorem 1
• proof
• Theorem 2