Simulations of Common Unsupervised Domain Adaptation Algorithms for Image Classification

TL;DR

This paper investigates unsupervised domain adaptation (UDA) for image classification by simulating and comparing a range of prevalent DA techniques on public datasets. It covers traditional, adversarial, and non-adversarial deep-learning approaches, emphasizing SSRT's strong Office-31 performance (83–92% range in simulations) and its sensitivity to batch size on Office-Home. Key contributions include a systematic cross-method evaluation, guidance on when to prefer certain backbones or training regimes, and insights into data quality, cross-dataset transfer, and medical-domain applicability. The study also discusses real-world challenges such as privacy constraints, the rise of foundation-model-based DA, self-supervised and test-time adaptation, and the need for robust OOD/domain drift handling, with public code available at the provided GitHub repository.

Abstract

Traditional machine learning assumes that training and test sets are derived from the same distribution; however, this assumption does not always hold in practical applications. This distribution disparity can lead to severe performance drops when the trained model is used in new data sets. Domain adaptation (DA) is a machine learning technique that aims to address this problem by reducing the differences between domains. This paper presents simulation-based algorithms of recent DA techniques, mainly related to unsupervised domain adaptation (UDA), where labels are available only in the source domain. Our study compares these techniques with public data sets and diverse characteristics, highlighting their respective strengths and drawbacks. For example, Safe Self-Refinement for Transformer-based DA (SSRT) achieved the highest accuracy (91.6\%) in the office-31 data set during our simulations, however, the accuracy dropped to 72.4\% in the Office-Home data set when using limited batch sizes. In addition to improving the reader's comprehension of recent techniques in DA, our study also highlights challenges and upcoming directions for research in this domain. The codes are available at https://github.com/AIPMLab/Domain_Adaptation.

Figures (7)

• Figure 1: Illustration of domain adaptation (DA). Left: feature distributions in source and target domains lead to misclassification of target domain samples by a model trained on source data. Right: DA reduces the discrepancy between source and target data distributions, enhancing the model's generalization performance.
• Figure 2: Bar graph illustrating the number of studies released on the topic of domain adaptation. The data was gathered from searches conducted on Google Scholar and PubMed databases.
• Figure 3: Timeline of domain adaptation techniques. To simplify, this illustration includes common DA methods based on traditional approaches (Green flag) and deep learning techniques (non-adversarial (Orange flag) and adversarial methods (Blue flag)). TCA: Transfer component analysis; MMD: Maximum mean discrepancy; JDA: Joint distribution adaptation; DDC: Deep domain confusion; DANN: Domain adversarial neural network; CycleGAN: Cycle-Consistent Adversarial Networks;DAAN: Dynamic adversarial adaptation networks; DAN: Deep adaptation network; ADDA: Adversarial discriminative domain adaptation; JAN: Joint adaptation network; CDAN: Conditional adversarial domain adaptation; DSAN: Deep subdomain adaptation network; DANN-IB: Generative feature replay for incremental domain adaptation; BNM: Batch nuclear-norm maximization; DALN: Discriminator-free adversarial learning network; SSRT: Safe self refinement for transformer based domain adaptation.
• Figure 4: The architecture proposed in ganin2016domain comprises a conventional feed-forward design with a label predictor (green) and a feature extractor $e_f$ (blue). The method of unsupervised domain adaptation integrates a domain classifier $D$ (Light salmon), which is connected to the feature extractor $e_f$ via a gradient reversal layer.
• Figure 5: Overview of SSRT SSRT. (Left) shows the Self-Refinement for their transformer-based model. Ptch Emb means Patch Embedding, and the two branches share the same parameters. Random offsets are added to the input token sequences of transformer (TF) blocks. The model is trained using its predictions of the original and perturbed versions. It is supervised by Kullback-Leibler divergence (a distance between two probabilities). (Right) shows the Safe Training mechanism.