On Causality in Domain Adaptation and Semi-Supervised Learning: an Information-Theoretic Analysis for Parametric Models

TL;DR

It is shown that in causal learning, the excess risk depends on the size of the source sample at a rate of $O(\frac{1}{m})$ only if the labelling distribution between the source and target domains remains unchanged, and in anti-causal learning, the unlabelled data dominate the performance at a rate of typically $O(\frac{1}{n})$.

Abstract

Recent advancements in unsupervised domain adaptation (UDA) and semi-supervised learning (SSL), particularly incorporating causality, have led to significant methodological improvements in these learning problems. However, a formal theory that explains the role of causality in the generalization performance of UDA/SSL is still lacking. In this paper, we consider the UDA/SSL scenarios where we access $m$ labelled source data and $n$ unlabelled target data as training instances under different causal settings with a parametric probabilistic model. We study the learning performance (e.g., excess risk) of prediction in the target domain from an information-theoretic perspective. Specifically, we distinguish two scenarios: the learning problem is called causal learning if the feature is the cause and the label is the effect, and is called anti-causal learning otherwise. We show that in causal learning, the excess risk depends on the size of the source sample at a rate of $O(\frac{1}{m})$ only if the labelling distribution between the source and target domains remains unchanged. In anti-causal learning, we show that the unlabelled data dominate the performance at a rate of typically $O(\frac{1}{n})$. These results bring out the relationship between the data sample size and the hardness of the learning problem with different causal mechanisms.

Figures (8)

• Figure 1: Causal settings for $X \rightarrow Y$ in (a) and $Y \rightarrow X$ in (b). We refer to the scenario in (a) as the "causal learning" setting because the direction of causation aligns with the direction of prediction, whereas the scenario in (b) is termed the "anti-causal learning" setting since the direction of causation is opposite to the direction of prediction.
• Figure 2: Excess risk comparisons under causal learning. (a) and (c) represents the results of $\mathcal{R}(b)$ for general shift case and concept drift learning, where we vary $n$ from 500 to 16000 in (a), and fix $n = 2000$ but vary $m$ from 500 to 16000 in (c). We sketch the results $\mathcal{R}(b)$ for covariate shift and semi-supervised learning in (b) and (d), here we fix $n = 2000$ and vary $m$ from 500 to 16000. We also plot $\mathcal{R}(b)^{-1}$ to show the rate w.r.t. $m$. We plot all excess risks in blue and their reciprocals in red. All results are derived by 3000 experimental repeats.
• Figure 3: Excess risk comparisons under anti-causal learning. (a) represents the results of $\mathcal{R}(b)$ and $\mathcal{R}(b)^{-1}$ for general shift case, and we vary $n$ from 500 to 16000. We sketch the results $\mathcal{R}(b)$ for label shift, label concept drift and semi-supervised learning in (b), (c) and (d). Here we fix $n = 2000$ and vary $m$ from 500 to 16000. It can be seen in that $\mathcal{R}(b)$ converges to a non-zero value $\lambda$ with $m$ increasing in (b) and (c), then we also plot $(\mathcal{R}(b) - \lambda)^{-1}$ to show the rate w.r.t. $m+n$. We plot all the excess risks in blue and their reciprocals in red. All results are derived by 3000 experimental repeats.
• Figure 4: Accuracy v.s. unlabelled sample size for digit pair (2, 5)
• Figure 5: Visualization of clusters for various source and target combinations for digit pair (2, 5) for initial and updated GMM with 100 labelled data and 500 unlabelled data

Theorems & Definitions (18)

• Definition 1: Causal Settings
• Remark 2
• Remark 3
• Definition 4: Log-loss
• Theorem 5: Excess Risk with Log-loss
• Theorem 6: Excess Risk with Exponential Concave Loss
• Theorem 7: Excess Risk with Bounded Loss
• Remark 8
• Remark 9
• Remark 10