Invariant Learning via Probability of Sufficient and Necessary Causes
Mengyue Yang, Zhen Fang, Yonggang Zhang, Yali Du, Furui Liu, Jean-Francois Ton, Jianhong Wang, Jun Wang
TL;DR
This work tackles out-of-distribution generalization by moving beyond invariant representations to capture the essential sufficiency and necessity of causal features. It introduces PNS risk as a principled objective to quantify the likelihood that a representation encodes both necessary and sufficient causes for the label, and it derives identifiability and generalization bounds under exogeneity and monotonicity. The authors propose CaSN, a learning framework that minimizes a worst-case PNS risk with semantic separability constraints while enforcing exogeneity through domain-appropriate penalties. Empirical results on synthetic and real-world benchmarks demonstrate improved OOD performance and more interpretable, causally meaningful representations, including strong generalization on PACS, VLCS, Colored MNIST, and SpuCo datasets. Overall, the approach provides a theoretically grounded and practically effective pathway for robust causal representation learning in the wild.
Abstract
Out-of-distribution (OOD) generalization is indispensable for learning models in the wild, where testing distribution typically unknown and different from the training. Recent methods derived from causality have shown great potential in achieving OOD generalization. However, existing methods mainly focus on the invariance property of causes, while largely overlooking the property of \textit{sufficiency} and \textit{necessity} conditions. Namely, a necessary but insufficient cause (feature) is invariant to distribution shift, yet it may not have required accuracy. By contrast, a sufficient yet unnecessary cause (feature) tends to fit specific data well but may have a risk of adapting to a new domain. To capture the information of sufficient and necessary causes, we employ a classical concept, the probability of sufficiency and necessary causes (PNS), which indicates the probability of whether one is the necessary and sufficient cause. To associate PNS with OOD generalization, we propose PNS risk and formulate an algorithm to learn representation with a high PNS value. We theoretically analyze and prove the generalizability of the PNS risk. Experiments on both synthetic and real-world benchmarks demonstrate the effectiveness of the proposed method. The details of the implementation can be found at the GitHub repository: https://github.com/ymy4323460/CaSN.