Why Domain Generalization Fail? A View of Necessity and Sufficiency

TL;DR

This work investigates domain generalization under limited training domains through a lens of necessary and sufficient conditions for generalization. It formalizes a causally informed framework, showing that conventional DG methods largely optimize sufficient conditions and can violate necessary ones, which explains their inconsistent gains over ERM. The authors introduce Subspace Representation Alignment (SRA), a practical method that preserves necessary conditions while enabling sufficiency by aligning representations within subspaces and using prototypes with Wasserstein clustering, achieving strong DG performance on DomainBed benchmarks. They connect ensemble strategies to the preservation of invariance information and demonstrate that these ideas can improve generalization when domain diversity is constrained. The work offers a principled path toward DG methods that exist and generalize reliably in realistic data regimes with few domains.

Abstract

Despite a strong theoretical foundation, empirical experiments reveal that existing domain generalization (DG) algorithms often fail to consistently outperform the ERM baseline. We argue that this issue arises because most DG studies focus on establishing theoretical guarantees for generalization under unrealistic assumptions, such as the availability of sufficient, diverse (or even infinite) domains or access to target domain knowledge. As a result, the extent to which domain generalization is achievable in scenarios with limited domains remains largely unexplored. This paper seeks to address this gap by examining generalization through the lens of the conditions necessary for its existence and learnability. Specifically, we systematically establish a set of necessary and sufficient conditions for generalization. Our analysis highlights that existing DG methods primarily act as regularization mechanisms focused on satisfying sufficient conditions, while often neglecting necessary ones. However, sufficient conditions cannot be verified in settings with limited training domains. In such cases, regularization targeting sufficient conditions aims to maximize the likelihood of generalization, whereas regularization targeting necessary conditions ensures its existence. Using this analysis, we reveal the shortcomings of existing DG algorithms by showing that, while they promote sufficient conditions, they inadvertently violate necessary conditions. To validate our theoretical insights, we propose a practical method that promotes the sufficient condition while maintaining the necessary conditions through a novel subspace representation alignment strategy. This approach highlights the advantages of preserving the necessary conditions on well-established DG benchmarks.

Figures (4)

• Figure 1: A directed acyclic graph (DAG) describing the causal relations among different factors producing data $X$ and label $Y$ in our SCM. Observed variables are shaded.
• Figure 2: Venn diagram of the optimal hypothesis spaces induced by a DG algorithm $A$.
• Figure 3: (Left) RandomResizedCrop alters the label-information, whereas ColorJitter does not. (Right) ColorJitter modifies the label-information of traffic light.
• Figure 4: Information diagrams of $X, Y$; the invariant representation $g_c(X)$; the minimal representation $g_{\text{min}}(X)$; and the representations $g_i(X), g_j(X)$, where there exist corresponding classifiers on top of these representations that form optimal hypotheses for the training domains.

Theorems & Definitions (35)

• Proposition 2.2
• Definition 3.1
• Theorem 3.2
• Definition 3.3
• Theorem 3.4
• Theorem 4.1
• Corollary 4.2
• Theorem 5.1
• Corollary 1.3
• proof