Unsupervised Domain Adaptation Using Compact Internal Representations

TL;DR

This work tackles unsupervised domain adaptation by introducing IMUDA, which enhances generalization under domain shift by enforcing larger interclass margins in the embedding space. It models the internally learned source distribution with a Gaussian Mixture Model and constructs a high-confidence pseudo-dataset to regularize the encoder so target-domain representations move away from class boundaries. The approach is supported by a PAC-style theoretical bound that links target error to source performance, SWD-based alignment to the pseudo-distribution, and the confidence level of pseudo-labels. Empirical results on four standard UDA benchmarks show competitive performance against strong baselines, with ablations confirming the contributions of both the pseudo-data alignment and the margin-enforcing components.

Abstract

A major technique for tackling unsupervised domain adaptation involves mapping data points from both the source and target domains into a shared embedding space. The mapping encoder to the embedding space is trained such that the embedding space becomes domain agnostic, allowing a classifier trained on the source domain to generalize well on the target domain. To further enhance the performance of unsupervised domain adaptation (UDA), we develop an additional technique which makes the internal distribution of the source domain more compact, thereby improving the model's ability to generalize in the target domain.We demonstrate that by increasing the margins between data representations for different classes in the embedding space, we can improve the model performance for UDA. To make the internal representation more compact, we estimate the internally learned multi-modal distribution of the source domain as Gaussian mixture model (GMM). Utilizing the estimated GMM, we enhance the separation between different classes in the source domain, thereby mitigating the effects of domain shift. We offer theoretical analysis to support outperofrmance of our method. To evaluate the effectiveness of our approach, we conduct experiments on widely used UDA benchmark UDA datasets. The results indicate that our method enhances model generalizability and outperforms existing techniques.

Figures (3)

• Figure 1: High-level description of the proposed unsupervised domain adaptation algorithm: Initially, a pretrained model on the source domain learns distinct and separable clusters in the embedding space, as shown in the top-left image. However, when this model is applied to the target domain, the model's performance deteriorates due to reduced class separability in the embedding space, as depicted in the bottom-left image. To address this issue, our algorithm generates a pseudo-dataset by selecting confident samples from the target domain. This pseudo-dataset is used to induce larger margins between the clusters representing different classes in the embedding space. By making each class cluster more compact, we create additional separation between them. Consequently, the model's generalizability improves, as illustrated in the top-right image. This enhanced generalizability aids in mitigating the effects of domain shift in the target domain, allowing for greater tolerance to domain variations before violating the margins between the classes, as depicted in the bottom-right image. Our algorithm leverages the pseudo-dataset and the concept of compactness in class clusters to improve the model's performance in the target domain, enabling mitigating the effect of domain shift.
• Figure 2: UMAP visualization for the representations of the dataset testing split for the $\mathcal{C}\rightarrow \mathcal{P}$ task: (a) the source domain (b) the target domain prior to adaptation, (c) samples drawn from the learned GMM, (d) the target domain after adaptation. (Best viewed enlarged on screen and in color).
• Figure 3: Empirical analysis based on the VisDA task: (a) loss function on the training split versus #epochs and (b) learning curve for the testing split versus #epochs; Effect of parameter values for the $\mathcal{C}\rightarrow \mathcal{P}$ task (c) performance versus the trade-off parameter $\lambda$ and (d) classification accuracy versus the confidence parameter $\tau$. (Best viewed enlarged on screen and in color).