Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gradual Domain Adaptation via Normalizing Flows

Shogo Sagawa, Hideitsu Hino

TL;DR

The paper tackles unsupervised domain adaptation under large source–target gaps where gradual self-training is ineffective due to few intermediate domains. It introduces a continuous normalizing flow framework that learns a continuous transformation from the target distribution toward a Gaussian mixture via the source domain, enabling gradual shifts without self-training. The approach combines CNFs with non-parametric log-likelihood estimation, a Gaussian mixture base, and a theoretical bound linking domain transport to target risk; experiments show improved performance and the ability to generate synthetic intermediate domains. This method offers a scalable, theoretically grounded alternative to self-training in challenging domain-shift settings, with practical implications for real-world tasks exhibiting substantial domain gaps.

Abstract

Standard domain adaptation methods do not work well when a large gap exists between the source and target domains. Gradual domain adaptation is one of the approaches used to address the problem. It involves leveraging the intermediate domain, which gradually shifts from the source domain to the target domain. In previous work, it is assumed that the number of intermediate domains is large and the distance between adjacent domains is small; hence, the gradual domain adaptation algorithm, involving self-training with unlabeled datasets, is applicable. In practice, however, gradual self-training will fail because the number of intermediate domains is limited and the distance between adjacent domains is large. We propose the use of normalizing flows to deal with this problem while maintaining the framework of unsupervised domain adaptation. The proposed method learns a transformation from the distribution of the target domain to the Gaussian mixture distribution via the source domain. We evaluate our proposed method by experiments using real-world datasets and confirm that it mitigates the above-explained problem and improves the classification performance.

Gradual Domain Adaptation via Normalizing Flows

TL;DR

The paper tackles unsupervised domain adaptation under large source–target gaps where gradual self-training is ineffective due to few intermediate domains. It introduces a continuous normalizing flow framework that learns a continuous transformation from the target distribution toward a Gaussian mixture via the source domain, enabling gradual shifts without self-training. The approach combines CNFs with non-parametric log-likelihood estimation, a Gaussian mixture base, and a theoretical bound linking domain transport to target risk; experiments show improved performance and the ability to generate synthetic intermediate domains. This method offers a scalable, theoretically grounded alternative to self-training in challenging domain-shift settings, with practical implications for real-world tasks exhibiting substantial domain gaps.

Abstract

Standard domain adaptation methods do not work well when a large gap exists between the source and target domains. Gradual domain adaptation is one of the approaches used to address the problem. It involves leveraging the intermediate domain, which gradually shifts from the source domain to the target domain. In previous work, it is assumed that the number of intermediate domains is large and the distance between adjacent domains is small; hence, the gradual domain adaptation algorithm, involving self-training with unlabeled datasets, is applicable. In practice, however, gradual self-training will fail because the number of intermediate domains is limited and the distance between adjacent domains is large. We propose the use of normalizing flows to deal with this problem while maintaining the framework of unsupervised domain adaptation. The proposed method learns a transformation from the distribution of the target domain to the Gaussian mixture distribution via the source domain. We evaluate our proposed method by experiments using real-world datasets and confirm that it mitigates the above-explained problem and improves the classification performance.
Paper Structure (39 sections, 3 theorems, 41 equations, 13 figures, 1 table, 1 algorithm)

This paper contains 39 sections, 3 theorems, 41 equations, 13 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related works
  3. Gradual domain adaptation
  4. Normalizing flows
  5. Formulation of gradual domain adaptation
  6. Proposed method
  7. Learning gradual shifts with normalizing flows
  8. Non-parametric estimation of log-likelihood
  9. Gaussian mixture model
  10. Theoretical aspects of flow-based model
  11. Scalability
  12. Experiments
  13. Datasets
  14. Experimental settings
  15. Learning gradual shifts with discrete normalizing flows
  16. ...and 24 more sections

Key Result

Proposition 1

If Assumption ass:logp holds, we have where $\mathrm{KL}[\cdot|\cdot]$ denotes the KL divergence and $D_{t}=\mathbb{E}_{p_{t+1}(\bm{x})}[\mathrm{KL}[p_{t+1}(y|\bm{x})|p_t(y|\bm{x})]]$.

Figures (13)

  • Figure 1: Overview of the proposed method. Owing to the limited number of available intermediate domains, the applicability of gradual self-training is limited. Gradual domain adaptation is possible without gradual self-training by utilizing continuous normalizing flow.
  • Figure 2: Comparison between the discrete and the continuous normalizing flows. Whereas continuous NFs are suitable for learning continuous change, discrete NFs are unsuitable for learning continuous change.
  • Figure 3: Comparison of the methods of estimating $\log p_{t\!-\!1}(g(\bm{x}^{(t)},t\!-\!1))$. The CNF trained with $k$NN estimators transforms the target data to the source data as expected
  • Figure 4: Experimental result of the training of our flow-based model with various hyperparameter $k$ values. The appropriate $k$ can be determined roughly by the fitting of the $k$NN classifier on the source dataset.
  • Figure 5: Number of dimensions is fixed.
  • ...and 8 more figures

Theorems & Definitions (6)

  • Proposition 1: nguyen2022kl
  • Proposition 2: onken2021ot
  • Corollary 1
  • proof
  • proof
  • proof