A Class-aware Optimal Transport Approach with Higher-Order Moment Matching for Unsupervised Domain Adaptation

TL;DR

CLOTH addresses unsupervised domain adaptation by introducing class-aware optimal transport that matches target data to source class-conditional regions in latent space via a cost $c\left(G(x),\mathbb{Q}_m^S\right)$. It leverages an amortized transportation network to predict transport probabilities and integrates class-aware Higher-Order Moment Matching (CaHoMM) for fine-grained distribution alignment. The framework combines a multi-class discriminator, adversarial training, and a suite of losses to jointly align distributions and preserve class structure, achieving state-of-the-art results on Digits, Office-31, Office-Home, and ImageCLEF-DA with efficient training. These contributions offer a scalable, principled approach to mitigating both data and label shifts in UDA, with practical impact for robust cross-domain learning.

Abstract

Unsupervised domain adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain. In this paper, we introduce a novel approach called class-aware optimal transport (OT), which measures the OT distance between a distribution over the source class-conditional distributions and a mixture of source and target data distribution. Our class-aware OT leverages a cost function that determines the matching extent between a given data example and a source class-conditional distribution. By optimizing this cost function, we find the optimal matching between target examples and source class-conditional distributions, effectively addressing the data and label shifts that occur between the two domains. To handle the class-aware OT efficiently, we propose an amortization solution that employs deep neural networks to formulate the transportation probabilities and the cost function. Additionally, we propose minimizing class-aware Higher-order Moment Matching (HMM) to align the corresponding class regions on the source and target domains. The class-aware HMM component offers an economical computational approach for accurately evaluating the HMM distance between the two distributions. Extensive experiments on benchmark datasets demonstrate that our proposed method significantly outperforms existing state-of-the-art baselines.

Figures (7)

• Figure 1: The OT distance between two distributions: $\mathbb{P}$ and $\mathcal{P}^{S}$. $\mathbb{P}$ consists of atoms representing the source and target examples $\mathbf{x}_{i}$, while $\mathcal{P}^{S}$ consists of atoms representing the source class-conditional distribution $\mathbb{P}_{m}^{S}$.
• Figure 2: The framework of our proposed CLOTH consists of four components: a weight-sharing generator $G$ for mapping the source and target data into the latent space, a source classifier $\mathcal{C}$, a transportation network $\mathcal{T}$, and a multi-class discriminator $\mathcal{D}$. The model is trained by minimizing component losses $\mathcal{L}^{C},\mathcal{L}^{G,S}+\mathcal{L}^{G,T},\mathcal{L}^{\mathcal{D}},\mathcal{L}^{t}$ and $\mathcal{L}^{ent}$. (a) The classification loss $\mathcal{L}^{C}$ is minimized to accurately classify the source data with labels, resulting in clear decision boundaries and well-clustered source samples. (b) The generator $G$ and the multi-class discriminator $\mathcal{D}$ are trained adversarially, with updates alternated between $\mathcal{L}^{G,S}+\mathcal{L}^{G,T}$ and $\mathcal{L}^{\mathcal{D}}$. Unlike previous adversarial methods, the source and target samples are mixed-up in a class-aware manner, as further discussed in the ablation study. (c) The transportation loss $\mathcal{L}^{t}$ is minimized to transport target samples to the source class-conditional distribution, approximately minimizing $\mathcal{W}_{c,\boldsymbol{\pi}}\left(\mathbb{Q},\mathcal{Q}^{S}\right)$ in (\ref{['eq:OT_Q']}). (d) For target samples lying close to the decision boundary, $\mathcal{T}$ is strengthened to provide a confident transportation probability by minimizing $\mathcal{L}^{ent}$. This ensures that these target samples are equally aligned to all source class regions. (e) Class-aware HMM is proposed to more accurately capture the complex distributions of the source and target domains in the latent space, and therefore the class-aware matching between the two domains is further improved.
• Figure 3: $\mathcal{W}_{c,\boldsymbol{\pi}}\left(\mathbb{Q},\mathcal{Q}^{S}\right)$ during the training on transfer tasks A$\rightarrow$W and P$\rightarrow$I.
• Figure 4: The t-SNE visualization with different scenarios of discriminator $\mathcal{D}$ on the transfer task A→ D. Each color represents a class, while the circle and cross markers indicate the source and target data, respectively.
• Figure 5: Analysis of twisting the $q$-order on transfer tasks A$\rightarrow$W (Office-31) and P$\rightarrow$I (ImageCLEF-DA).

Theorems & Definitions (1)

• Theorem 1