Out-of-Variable Generalization for Discriminative Models

TL;DR

This work tackles out-of-variable (OOV) generalization, where test environments reveal variables never jointly observed with the target during training. It shows that mere marginal consistency between source and target discriminative models cannot identify the optimal target predictor, proving an impossibility result. The authors propose MomentLearn, a residual-moment based method that uses the third central moment of the source-model residuals to infer the unobserved causal derivative $\partial \phi/\partial X_3$ and construct a zero-shot target predictor that surpasses naive baselines. Theoretical results establish when residual moments yield identifiability (e.g., certain additive-noise causal forms) and a practical algorithm is demonstrated on synthetic data and a real-world mtcars dataset, illustrating nontrivial OOV transfer and robustness under mild assumption violations. This approach highlights a principled way to leverage latent information in residuals for generalization across environments with differing variable observability, suggesting avenues for integrating OOD and OOV techniques in real-world applications.

Abstract

The ability of an agent to do well in new environments is a critical aspect of intelligence. In machine learning, this ability is known as $\textit{strong}$ or $\textit{out-of-distribution}$ generalization. However, merely considering differences in data distributions is inadequate for fully capturing differences between learning environments. In the present paper, we investigate $\textit{out-of-variable}$ generalization, which pertains to an agent's generalization capabilities concerning environments with variables that were never jointly observed before. This skill closely reflects the process of animate learning: we, too, explore Nature by probing, observing, and measuring $\textit{subsets}$ of variables at any given time. Mathematically, $\textit{out-of-variable}$ generalization requires the efficient re-use of past marginal information, i.e., information over subsets of previously observed variables. We study this problem, focusing on prediction tasks across environments that contain overlapping, yet distinct, sets of causes. We show that after fitting a classifier, the residual distribution in one environment reveals the partial derivative of the true generating function with respect to the unobserved causal parent in that environment. We leverage this information and propose a method that exhibits non-trivial out-of-variable generalization performance when facing an overlapping, yet distinct, set of causal predictors.

Figures (5)

• Figure 1:
• Figure 3: Examples of OOV scenarios where marginal consistency condition alone (\ref{['eqref:joint-to-marginal']}) does not permit the identification of the optimal predictive function in the corresponding target domain.
• Figure 4: Shown are mean of the relative loss (and its $95\%$ confidence interval) for varying numbr of joint samples observed in the target domain. MomentLearn outperforms the joint predictor in the few-sample region and is always preferred over the marginal predictor.
• Figure 5: Examples of OOV scenarios where marginal consistency condition alone (\ref{['eqref:joint-to-marginal']}) does not permit the identification of the optimal predictive function in the corresponding target domain.
• Figure 6: An example of the scenarios considered in Theorem \ref{['thm:possibility']}

Theorems & Definitions (17)

• Definition 1: OOV Generalization
• Theorem 1
• Theorem 2
• Theorem 3
• Corollary 3
• Theorem 3
• Theorem 4
• proof
• Theorem 4
• Theorem 5