A DIRT-T Approach to Unsupervised Domain Adaptation

TL;DR

This paper addresses unsupervised, non-conservative domain adaptation by leveraging the cluster assumption to constrain decision boundaries away from high-density regions. It introduces VADA, which augments domain adversarial training with conditional entropy minimization and virtual adversarial training, and DIRT-T, which refines VADA using natural-gradient steps and a teacher-guided KL constraint. Empirical results across diverse visual and non-visual domains show state-of-the-art performance and robust ablations confirm the importance of VAT, entropy minimization, and the teacher-driven refinement. The work demonstrates that incorporating cluster-based priors into deep domain adaptation yields substantial improvements and broad applicability, with potential extensions to weakly supervised settings and enhanced optimization techniques.

Abstract

Domain adaptation refers to the problem of leveraging labeled data in a source domain to learn an accurate model in a target domain where labels are scarce or unavailable. A recent approach for finding a common representation of the two domains is via domain adversarial training (Ganin & Lempitsky, 2015), which attempts to induce a feature extractor that matches the source and target feature distributions in some feature space. However, domain adversarial training faces two critical limitations: 1) if the feature extraction function has high-capacity, then feature distribution matching is a weak constraint, 2) in non-conservative domain adaptation (where no single classifier can perform well in both the source and target domains), training the model to do well on the source domain hurts performance on the target domain. In this paper, we address these issues through the lens of the cluster assumption, i.e., decision boundaries should not cross high-density data regions. We propose two novel and related models: 1) the Virtual Adversarial Domain Adaptation (VADA) model, which combines domain adversarial training with a penalty term that punishes the violation the cluster assumption; 2) the Decision-boundary Iterative Refinement Training with a Teacher (DIRT-T) model, which takes the VADA model as initialization and employs natural gradient steps to further minimize the cluster assumption violation. Extensive empirical results demonstrate that the combination of these two models significantly improve the state-of-the-art performance on the digit, traffic sign, and Wi-Fi recognition domain adaptation benchmarks.

Figures (7)

• Figure 1: VADA improves upon domain adversarial training by additionally penalizing violations of the cluster assumption.
• Figure 2: DIRT-T uses VADA as initialization. After removing the source training signal, DIRT-T minimizes cluster assumption violation in the target domain through a series of natural gradient steps.
• Figure 3: Effect of applying instance normalization to the input image. In clockwise direction: MNIST-M, GTSRB, SVHN, and CIFAR-10. In each quadrant, the top row is the original image, and the bottom row is the instance-normalized image.
• Figure 4: Comparing model behavior with and without the application of the KL-term. At iteration 0, we begin with the VADA initialization and apply the DIRT-T algorithm.
• Figure 5: T-SNE plot of the last hidden layer for MNIST (blue) $\to$ SVHN (red). We used the model without instance normalization to highlight the further improvement that DIRT-T provides.

Theorems & Definitions (1)

• Theorem 1