Towards Trustworthy Unsupervised Domain Adaptation: A Representation Learning Perspective for Enhancing Robustness, Discrimination, and Generalization

TL;DR

This work tackles robust unsupervised domain adaptation by reframing RoUDA through mutual information theory. It introduces MIRoUDA, a MI-based framework that enforces robust (minimizing $I(X;F)$), discriminative (maximizing $I(F;Y)$), and generalized (minimizing $I(F_1;F_2)$) representations within a self-training pipeline using a dual-model architecture and a consensus regularizer. The method demonstrates superior robustness across multiple UDA benchmarks while preserving or improving clean accuracy, outperforming both baselines and state-of-the-art RoUDA methods. The combination of MI-driven objectives, dual-model diversification, and consensus regularization offers a principled path to robust, discriminative, and generalizable domain adaptation with practical impact on real-world deployments.

Abstract

Robust Unsupervised Domain Adaptation (RoUDA) aims to achieve not only clean but also robust cross-domain knowledge transfer from a labeled source domain to an unlabeled target domain. A number of works have been conducted by directly injecting adversarial training (AT) in UDA based on the self-training pipeline and then aiming to generate better adversarial examples (AEs) for AT. Despite the remarkable progress, these methods only focus on finding stronger AEs but neglect how to better learn from these AEs, thus leading to unsatisfied results. In this paper, we investigate robust UDA from a representation learning perspective and design a novel algorithm by utilizing the mutual information theory, dubbed MIRoUDA. Specifically, through mutual information optimization, MIRoUDA is designed to achieve three characteristics that are highly expected in robust UDA, i.e., robustness, discrimination, and generalization. We then propose a dual-model framework accordingly for robust UDA learning. Extensive experiments on various benchmarks verify the effectiveness of the proposed MIRoUDA, in which our method surpasses the state-of-the-arts by a large margin.

Figures (7)

• Figure 1: Different UDA schemes and their performance. Left: Traditional UDA methods do not take robustness into account, resulting in defenseless against adversarial attacks. Middle: Existing RoUDA can improve the robustness via incorporating AT but the results are under-optimal due to their empirical design. Right: Our MIRoUDA unifies the MI theory for improving robustness, discrimination, and generalization. We use CDAN long2018conditional as the UDA baseline, and PGD-20 madry2018towards for evaluating model robustness. The results of RoUDA are from the SRoUDA zhu2022srouda proposed in 2023.
• Figure 2: Overview of our proposed MIRoUDA. The whole framework is based on the self-training pipeline, where a source model $G_s$ is first pre-trained by the UDA baseline. We omit this part due to its commonplace. Here we put emphasis on the robust target training part, where two dual models are trained parallelly for MI optimization. We use the MI objective $\min I(X;F)$, $\max I(F;Y)$, and $\min I(F_1;F_2)$ to achieve robust, discriminative, and generalized representation learning, respectively. A consensus regularizer is equipped to force the output consensus between the source and robust model on clean target data, which consequently helps in preserving high clean accuracy.
• Figure 3: Robust accuracy(%) of ResNet50 trained with $l_\infty$ of $\epsilon=8/255$ boundary against unseen attacks. For unseen attacks, we use PGD-50 under different-sized $l_\infty$ balls, and other types of norm balls, e.g., $l_1$, and $l_2$.
• Figure 4: The effect of hyper-parameters $\alpha$ and $\beta$ on model performance in clean accuracy (left) and robust accuracy (right). The stable performance shows its robustness against hyper-parameters.
• Figure 5: (a)-(d): Visualization of features using t-SNE embeddings NIPS2002_6150ccc6 from models trained with different methods on A$\rightarrow$W task. The red dots stand for clean target data, and the blue dots stand for adversarial examples of target data.