Unified Source-Free Domain Adaptation

TL;DR

This work tackles the practical challenge of Source-Free Domain Adaptation (SFDA) across Closed-set, Open-set, Partial-set, and Generalized settings by proposing Unified SFDA. It introduces CausalDA, a causality-guided method that discovers latent external and internal causal factors in the logit space, leveraging a frozen Vision-Language model (CLIP) and a self-supervised information bottleneck to achieve robust cross-domain adaptation without target-domain supervision. The approach yields state-of-the-art results across diverse SFDA benchmarks and demonstrates strong generalization to source-free OOD scenarios, supported by extensive ablations and analyses. The framework promises improved deployability in privacy-sensitive or data-restricted environments by relying on causal factors rather than solely statistical associations.

Abstract

In the pursuit of transferring a source model to a target domain without access to the source training data, Source-Free Domain Adaptation (SFDA) has been extensively explored across various scenarios, including Closed-set, Open-set, Partial-set, and Generalized settings. Existing methods, focusing on specific scenarios, not only address a limited subset of challenges but also necessitate prior knowledge of the target domain, significantly limiting their practical utility and deployability. In light of these considerations, we introduce a more practical yet challenging problem, termed unified SFDA, which comprehensively incorporates all specific scenarios in a unified manner. In this paper, we propose a novel approach latent Causal factors discovery for unified SFDA (CausalDA). In contrast to previous alternatives that emphasize learning the statistical description of reality, we formulate CausalDA from a causality perspective. The objective is to uncover potential causality between latent variables and model decisions, enhancing the reliability and robustness of the learned model against domain shifts. To integrate extensive world knowledge, we leverage a pre-trained vision-language model such as CLIP. This aids in the formation and discovery of latent causal factors in the absence of supervision in the variation of distribution and semantics, coupled with a newly designed information bottleneck with theoretical guarantees. Extensive experiments demonstrate that CausalDA can achieve new state-of-the-art results in distinct SFDA settings, as well as source-free out-of-distribution generalization. Our code and data are available at https://github.com/tntek/CausalDA.

Figures (9)

• Figure 1: Comparison of Structural Causal Models (SCM). (a) SCM for transfer learning with source domain, where $S$ and $U$ represent causal and non-causal factors. Domain shift is generally caused by $U$, which, together with the generalizable information $S$, e.g., the shape/structure of a dog (see Fig. \ref{['fig:domain-shift']}), form the observation $x$. (b) Our proposed SCM for SFDA in a latent space (e.g., the logit $\boldsymbol{a}$) where the causal factors $S$ are decomposed into the external $S_e$ and internal $S_{i}$ components.
• Figure 2: Illustration of domain shift. (a) Different appearance styles of dog; (b) Spurious dependence of background: Cow in grass ground vs. on a car.
• Figure 3: Overview of our CausalDA framework: (a) Phase-1: Discovering the external causal factors $S_e$ in form of prompt context $\boldsymbol{v}_{e}$ from a frozen ViL model using our self-supervised information bottleneck algorithm; (b) Phase-2: Discovering the internal causal factors $S_i$ where the updated prompt context $\boldsymbol{v}_{e}$ is used to predict pseudo-labels as the prior information.
• Figure 4: Realizing our self-supervised information bottleneck w.r.t. Theorem\ref{['thm-one']}. $\rm{T}_1$ is estimated by Variational Mutual Information (VMI) with Gaussian distribution assumption $\mathcal{N}(Z, \Sigma_Z)$. $\rm{T}_2$ is computed by Probabilistic Mutual Information (PMI) where function $f_w$ integrates min-max optimization, using covariance matrix $\Sigma_Z$ from $\rm{T}_1$ as weighting parameters $w$.
• Figure 5: Causality validation: Invariance analysis across varying noise settings on the Ar$\rightarrow$Cl task of Office-home.

Theorems & Definitions (2)

• Lemma 1
• Theorem 1