The Risks of Invariant Risk Minimization

TL;DR

This paper provides the first formal analysis of Invariant Risk Minimization (IRM) for classification under a latent-variable SEM separating invariant and environment-dependent features. It establishes a sharp linear-regime threshold: when the number of training environments E exceeds the environmental dimension d_e, the invariant predictor is optimal; otherwise, non-invariant environmental features can dominate training performance, hindering generalization. In the non-linear regime, IRM can fail entirely unless the training environments adequately cover potential test distributions, often performing no better than ERM. Collectively, the results challenge the universal efficacy of IRM and related invariant objectives in high-dimensional, non-linear settings and highlight the need for stronger theoretical guarantees.

Abstract

Invariant Causal Prediction (Peters et al., 2016) is a technique for out-of-distribution generalization which assumes that some aspects of the data distribution vary across the training set but that the underlying causal mechanisms remain constant. Recently, Arjovsky et al. (2019) proposed Invariant Risk Minimization (IRM), an objective based on this idea for learning deep, invariant features of data which are a complex function of latent variables; many alternatives have subsequently been suggested. However, formal guarantees for all of these works are severely lacking. In this paper, we present the first analysis of classification under the IRM objective--as well as these recently proposed alternatives--under a fairly natural and general model. In the linear case, we show simple conditions under which the optimal solution succeeds or, more often, fails to recover the optimal invariant predictor. We furthermore present the very first results in the non-linear regime: we demonstrate that IRM can fail catastrophically unless the test data are sufficiently similar to the training distribution--this is precisely the issue that it was intended to solve. Thus, in this setting we find that IRM and its alternatives fundamentally do not improve over standard Empirical Risk Minimization.

Figures (4)

• Figure 3.1: A Bayesian network depicting our model. Shading indicates the variable is observed.
• Figure C.1: Performance of predictors learned with IRM (5 different runs) and ERM (dashed lines) on test distributions where the correlation between environmental features and the label is consistent (no shift) or reversed (shift). The dashed green line is the performance of the optimal invariant predictor. Observe that up until $E = d_e$, IRM consistently returns a predictor with performance similar to ERM: good generalization without distribution shift, but catastrophic failure when the correlation is reversed. In contrast, once $E > d_e$, IRM is able to recover a $\Phi,\hat{\beta}$ with performance similar to that of the invariant optimal predictor.
• Figure C.2: Simulations to evaluate $\sigma_e\tilde{\mu}$ for varying ratios of $\frac{d_e}{d_c}$. When $\sigma_e^2=1$ the value closely tracks $\sqrt{d_e-E}$, and the crossover point is approximately $d_e - \sigma_e^2d_c$. These results imply the conditions of Theorem \ref{['thm:corr-less-01-risk']} are very likely to hold in the high-dimensional setting.
• Figure E.1: Simulations to evaluate $\tilde{\mu}$ for varying ratios of $\frac{d_e}{d_c}$. When $\sigma_e^2 = 1$ the value closely tracks $\sqrt{d_e-E}$, and the crossover point is approximately $d_e - \sigma_e^2d_c$. Due to the similarity of Equation \ref{['eq:project-sigma']} to Equation \ref{['eq:project-means-to-single1']}, it makes sense that the results are very similar to those presented in Figure \ref{['fig:tildemu-simulations']}.

Theorems & Definitions (42)

• Definition 1
• Theorem 1: Informal, Linear
• Theorem 2: Informal, Linear
• Theorem 3: Informal, Non-linear
• Proposition 4
• Theorem 5: Linear case
• Corollary 6
• proof : Proof Sketch
• Theorem 7
• Definition 2