Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Variance Matters: Improving Domain Adaptation via Stratified Sampling

Andrea Napoli, Paul White

TL;DR

Domain shift causes high-variance estimates of domain-discrepancy losses in unsupervised domain adaptation. The paper introduces VaRDASS, a variance-reduced stratified-sampling framework for UDA losses, with specialized stratification objectives for MMD and CORAL and a practical kernel-k-means–style optimizer. It provides theoretical variance bounds and demonstrates that variance reduction translates into better target-domain performance on three benchmarks. Overall, VaRDASS offers a scalable, principled approach to stabilize UDA training and improve generalization under domain shift.

Abstract

Domain shift remains a key challenge in deploying machine learning models to the real world. Unsupervised domain adaptation (UDA) aims to address this by minimising domain discrepancy during training, but the discrepancy estimates suffer from high variance in stochastic settings, which can stifle the theoretical benefits of the method. This paper proposes Variance-Reduced Domain Adaptation via Stratified Sampling (VaRDASS), the first specialised stochastic variance reduction technique for UDA. We consider two specific discrepancy measures -- correlation alignment and the maximum mean discrepancy (MMD) -- and derive ad hoc stratification objectives for these terms. We then present expected and worst-case error bounds, and prove that our proposed objective for the MMD is theoretically optimal (i.e., minimises the variance) under certain assumptions. Finally, a practical k-means style optimisation algorithm is introduced and analysed. Experiments on three domain shift datasets demonstrate improved discrepancy estimation accuracy and target domain performance.

Variance Matters: Improving Domain Adaptation via Stratified Sampling

TL;DR

Domain shift causes high-variance estimates of domain-discrepancy losses in unsupervised domain adaptation. The paper introduces VaRDASS, a variance-reduced stratified-sampling framework for UDA losses, with specialized stratification objectives for MMD and CORAL and a practical kernel-k-means–style optimizer. It provides theoretical variance bounds and demonstrates that variance reduction translates into better target-domain performance on three benchmarks. Overall, VaRDASS offers a scalable, principled approach to stabilize UDA training and improve generalization under domain shift.

Abstract

Domain shift remains a key challenge in deploying machine learning models to the real world. Unsupervised domain adaptation (UDA) aims to address this by minimising domain discrepancy during training, but the discrepancy estimates suffer from high variance in stochastic settings, which can stifle the theoretical benefits of the method. This paper proposes Variance-Reduced Domain Adaptation via Stratified Sampling (VaRDASS), the first specialised stochastic variance reduction technique for UDA. We consider two specific discrepancy measures -- correlation alignment and the maximum mean discrepancy (MMD) -- and derive ad hoc stratification objectives for these terms. We then present expected and worst-case error bounds, and prove that our proposed objective for the MMD is theoretically optimal (i.e., minimises the variance) under certain assumptions. Finally, a practical k-means style optimisation algorithm is introduced and analysed. Experiments on three domain shift datasets demonstrate improved discrepancy estimation accuracy and target domain performance.

Paper Structure

This paper contains 16 sections, 7 theorems, 21 equations, 5 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Method
  3. Preliminaries
  4. Optimal stratification for MMD
  5. Optimal stratification for CORAL
  6. Optimisation algorithm
  7. Overall training strategy
  8. Experiments
  9. Effect of $T$
  10. Limitations
  11. Conclusion
  12. Acknowledgements
  13. Proof of Theorem \ref{['theorem1']}
  14. Proof of Theorem \ref{['theorem4']}
  15. Performance breakdown by data split
  16. ...and 1 more sections

Key Result

Lemma 1

Assume ${\widehat{\mu}}_{\phi,s}\sim\mathcal{N}\left( \mu_{\phi,s},\Sigma_{{\widehat{\mu}}_{\phi,s}} \right)$ and ${\widehat{\mu}}_{\phi,t}\sim \mathcal{N}\left( \mu_{\phi,t},\Sigma_{{\widehat{\mu}}_{\phi,t}} \right)$ are independent random vectors, and that the relevant conditions for normality und

Figures (5)

  • Figure 1: The relationship between the target ($\mathrm{Var}\left( {\widehat{L}}_{\operatorname{MMD}} \right)$) and surrogate ($\mathrm{Var}\left( {\widehat{\mu}}_{\phi,s} \right)$) partitioning objectives, for a range of feature dimensionalities $d$.
  • Figure 2: The relationship between the target ($\mathrm{Var}\left( {\widehat{R}}_{s} \right)$) and surrogate ($\mathrm{Var}\left( {\widehat{R}}_{s}' \right)$) partitioning objectives, for a range of feature dimensionalities $d$, and two distributions.
  • Figure 3: The performance characteristics of Algorithm \ref{['alg2']}.
  • Figure 4: Estimator variance vs minibatch size for Algorithm \ref{['alg1']} (purple) vs 3 ablations.
  • Figure 5: Estimator variance ($\mathrm{Var}\left( {\widehat{\mu}}_{\phi,s} \right)$) throughout training with VaRDASS for different values of $T$, also compared to uniform random sampling.

Theorems & Definitions (13)

  • Lemma 1
  • Theorem 1
  • proof
  • Theorem 2: Worst-case error bound
  • proof
  • Theorem 3: Expected error
  • proof
  • Theorem 4
  • proof
  • Theorem 1
  • ...and 3 more