Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distributionally Robust Classification for Multi-source Unsupervised Domain Adaptation

Seonghwi Kim, Sung Ho Jo, Wooseok Ha, Minwoo Chae

TL;DR

This work tackles distribution shifts in unsupervised domain adaptation by proposing a distributionally robust optimization framework that models uncertainty in both the target covariate distribution and the conditional label distribution. It introduces a multi-source-inspired ambiguity set that mixtures source conditionals while allowing covariate perturbations in the target space, and provides a tractable minimax algorithm with alternating updates for the feature map, mixture weights, and classifier parameters. The approach integrates with existing UDA methods and demonstrates substantial improvements on digit-recognition benchmarks and spurious-correlation datasets, particularly when target data are scarce or spurious cues are prevalent. The framework offers robust generalization across diverse domain shifts and can be extended to single-source settings via pseudo-sources, broadening its practical impact.

Abstract

Unsupervised domain adaptation (UDA) is a statistical learning problem when the distribution of training (source) data is different from that of test (target) data. In this setting, one has access to labeled data only from the source domain and unlabeled data from the target domain. The central objective is to leverage the source data and the unlabeled target data to build models that generalize to the target domain. Despite its potential, existing UDA approaches often struggle in practice, particularly in scenarios where the target domain offers only limited unlabeled data or spurious correlations dominate the source domain. To address these challenges, we propose a novel distributionally robust learning framework that models uncertainty in both the covariate distribution and the conditional label distribution. Our approach is motivated by the multi-source domain adaptation setting but is also directly applicable to the single-source scenario, making it versatile in practice. We develop an efficient learning algorithm that can be seamlessly integrated with existing UDA methods. Extensive experiments under various distribution shift scenarios show that our method consistently outperforms strong baselines, especially when target data are extremely scarce.

Distributionally Robust Classification for Multi-source Unsupervised Domain Adaptation

TL;DR

This work tackles distribution shifts in unsupervised domain adaptation by proposing a distributionally robust optimization framework that models uncertainty in both the target covariate distribution and the conditional label distribution. It introduces a multi-source-inspired ambiguity set that mixtures source conditionals while allowing covariate perturbations in the target space, and provides a tractable minimax algorithm with alternating updates for the feature map, mixture weights, and classifier parameters. The approach integrates with existing UDA methods and demonstrates substantial improvements on digit-recognition benchmarks and spurious-correlation datasets, particularly when target data are scarce or spurious cues are prevalent. The framework offers robust generalization across diverse domain shifts and can be extended to single-source settings via pseudo-sources, broadening its practical impact.

Abstract

Unsupervised domain adaptation (UDA) is a statistical learning problem when the distribution of training (source) data is different from that of test (target) data. In this setting, one has access to labeled data only from the source domain and unlabeled data from the target domain. The central objective is to leverage the source data and the unlabeled target data to build models that generalize to the target domain. Despite its potential, existing UDA approaches often struggle in practice, particularly in scenarios where the target domain offers only limited unlabeled data or spurious correlations dominate the source domain. To address these challenges, we propose a novel distributionally robust learning framework that models uncertainty in both the covariate distribution and the conditional label distribution. Our approach is motivated by the multi-source domain adaptation setting but is also directly applicable to the single-source scenario, making it versatile in practice. We develop an efficient learning algorithm that can be seamlessly integrated with existing UDA methods. Extensive experiments under various distribution shift scenarios show that our method consistently outperforms strong baselines, especially when target data are extremely scarce.
Paper Structure (38 sections, 2 theorems, 24 equations, 7 figures, 2 tables, 1 algorithm)

This paper contains 38 sections, 2 theorems, 24 equations, 7 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related works
  3. Preliminaries
  4. Proposed distributionally robust learning method
  5. Multi-source framework for single-source problems
  6. Proposed ambiguity set: high-level formulation
  7. Detailed specification of the ambiguity set
  8. Learning algorithm
  9. 1. Update of $z'$.
  10. 2. Update of $\beta$.
  11. 3. Update of $\theta$.
  12. Experiments
  13. MNIST, SVHN, and USPS
  14. Waterbirds, CelebA, and CMNIST
  15. Experimental setup
  16. ...and 23 more sections

Key Result

Proposition 3.1

Suppose that the feature map $z: \mathcal{X} \to \mathcal{Z}$ and the estimators $\hat{P}_{Y \mid X}^{(k)}$ are given. Let the divergence measures $D_1$ and $D_2$ be defined as in Section ssec:details. For any $\epsilon_1, \epsilon_2 >0$ and a fixed $\theta$, the maximization objective in the DRO pr where is the soft pseudo-label vector defined as a convex combination of source conditionals.

Figures (7)

  • Figure 1: Heatmaps of average test accuracy across ($\epsilon_1$,$\epsilon_2$).
  • Figure 2: Gradient behavior of $\beta$ and $\theta$ under joint optimization.
  • Figure 3: Heatmaps of average accuracy for the MNIST$\rightarrow$USPS task with $10^2$ unlabeled target samples per class, shown across different values of $K$ and $\epsilon_2$. Panels (a) and (b) show validation accuracy obtained using a small labeled target set and cross-validation, respectively. Panel (c) shows the corresponding test accuracy. All results are averaged over 5 independent runs.
  • Figure 4: Example images from the CMNIST dataset.
  • Figure 5: Example images from the Waterbirds dataset.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Proposition 3.1: Tractable surrogate reformulation
  • Lemma A.1.1: staib2017distributionally, Proposition 3.1