Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Out-of-Distribution Optimality of Invariant Risk Minimization

Shoji Toyota, Kenji Fukumizu

TL;DR

This paper rigorously proves that a solution to the bi-level optimization problem minimizes the o.o.d.risk, which is the maximum risk among all domains, and provides sufficient conditions on distributions providing training data and on a dimension of feature space for theBi-leveled optimization problem to minimize the o-o-d.

Abstract

Deep Neural Networks often inherit spurious correlations embedded in training data and hence may fail to generalize to unseen domains, which have different distributions from the domain to provide training data. M. Arjovsky et al. (2019) introduced the concept out-of-distribution (o.o.d.) risk, which is the maximum risk among all domains, and formulated the issue caused by spurious correlations as a minimization problem of the o.o.d. risk. Invariant Risk Minimization (IRM) is considered to be a promising approach to minimize the o.o.d. risk: IRM estimates a minimum of the o.o.d. risk by solving a bi-level optimization problem. While IRM has attracted considerable attention with empirical success, it comes with few theoretical guarantees. Especially, a solid theoretical guarantee that the bi-level optimization problem gives the minimum of the o.o.d. risk has not yet been established. Aiming at providing a theoretical justification for IRM, this paper rigorously proves that a solution to the bi-level optimization problem minimizes the o.o.d. risk under certain conditions. The result also provides sufficient conditions on distributions providing training data and on a dimension of feature space for the bi-leveled optimization problem to minimize the o.o.d. risk.

Out-of-Distribution Optimality of Invariant Risk Minimization

TL;DR

This paper rigorously proves that a solution to the bi-level optimization problem minimizes the o.o.d.risk, which is the maximum risk among all domains, and provides sufficient conditions on distributions providing training data and on a dimension of feature space for theBi-leveled optimization problem to minimize the o-o-d.

Abstract

Deep Neural Networks often inherit spurious correlations embedded in training data and hence may fail to generalize to unseen domains, which have different distributions from the domain to provide training data. M. Arjovsky et al. (2019) introduced the concept out-of-distribution (o.o.d.) risk, which is the maximum risk among all domains, and formulated the issue caused by spurious correlations as a minimization problem of the o.o.d. risk. Invariant Risk Minimization (IRM) is considered to be a promising approach to minimize the o.o.d. risk: IRM estimates a minimum of the o.o.d. risk by solving a bi-level optimization problem. While IRM has attracted considerable attention with empirical success, it comes with few theoretical guarantees. Especially, a solid theoretical guarantee that the bi-level optimization problem gives the minimum of the o.o.d. risk has not yet been established. Aiming at providing a theoretical justification for IRM, this paper rigorously proves that a solution to the bi-level optimization problem minimizes the o.o.d. risk under certain conditions. The result also provides sufficient conditions on distributions providing training data and on a dimension of feature space for the bi-leveled optimization problem to minimize the o.o.d. risk.
Paper Structure (28 sections, 6 theorems, 89 equations, 1 figure)

This paper contains 28 sections, 6 theorems, 89 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Main Results
  3. Case I: Least Square Loss
  4. Condition (i):
  5. Conditions (ii) and (iii):
  6. Condition (iv):
  7. Case II: Cross Entropy Loss
  8. Novelty and Significance of Theorems 1 and 2
  9. Setting of Domains
  10. Assumption on the Underlying Distribution $P_{Y^I|X_1^I}$
  11. Characterization of Invariance
  12. Range of Invariance
  13. Previous Works
  14. Proofs
  15. Proof Sketch of Main Theorems
  16. ...and 13 more sections

Key Result

Theorem 1

Domains $\{ (X^e, Y^e)\}_{e \in {\mathcal{E}}}$ are assumed to be (theo:assump1). We also assume that the following four conditions hold: Then, we have Here, ${\mathcal{F}}$ is the set of all measurable functions $f: {\mathcal{X}} \rightarrow {\mathcal{Y}}$.

Figures (1)

  • Figure 1: Supports of probability distributions $P_{X^a, Y^a}$ and $P_{X^b, Y^b}$. The figure implies that $P_{X^a, Y^a}(N_{y^*} \times \Phi^{-1} (\Phi^*)) \neq 0$ and $P_{X^b, Y^b}(N_{y^*} \times \Phi^{-1} (\Phi^*)) = 0$ ($\because (x_1^*, x_2^{**}) \notin \Phi^{-1} (\Phi^*)$ (\ref{['eq:assumption_contrudiction']})), and that $P_{X^a, Y^a}(\Phi^{-1} (\Phi^*)) \neq 0$ and $P_{X^b, Y^b}(\Phi^{-1} (\Phi^*)) \neq 0$. These Eqs. lead us $P_{Y^{a} |\Phi(X^{a})} (N_{y^*} | \Phi^*) \neq 0 = P_{Y^{b} |\Phi(X^{b})} (N_{y^*} | \Phi^*)$.

Theorems & Definitions (6)

  • Theorem 1: o.o.d. optimality of the bi-leveled optimization problem (\ref{['eq:IRM_conti']}) under least square loss setting
  • Theorem 2: o.o.d. optimality of the bi-leveled optimization problem (\ref{['eq:IRM_conti']}) under cross-entropy loss setting
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • Lemma 6