Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decorr: Environment Partitioning for Invariant Learning and OOD Generalization

Yufan Liao, Qi Wu, Zhaodi Wu, Xing Yan

TL;DR

Decorr tackles the challenge of improving invariant learning when natural environment labels are unavailable by algorithmically partitioning data into multiple low-correlation environments. It optimizes sample weights to minimize the weighted correlation distance to the identity, then applies IRM across the resulting environments to induce invariant predictors that generalize under covariate shifts and spurious correlations. Across synthetic, tabular, and image datasets, Decorr+IRM (and Decorr+REx) consistently outperforms ERM and other partitioning strategies, especially under model misspecification or distributional shifts. This approach broadens the applicability of invariant learning by eliminating reliance on predefined environments and offers practical gains in OOD robustness for diverse applications.

Abstract

Invariant learning methods, aimed at identifying a consistent predictor across multiple environments, are gaining prominence in out-of-distribution (OOD) generalization. Yet, when environments aren't inherent in the data, practitioners must define them manually. This environment partitioning--algorithmically segmenting the training dataset into environments--crucially affects invariant learning's efficacy but remains underdiscussed. Proper environment partitioning could broaden the applicability of invariant learning and enhance its performance. In this paper, we suggest partitioning the dataset into several environments by isolating low-correlation data subsets. Through experiments with synthetic and real data, our Decorr method demonstrates superior performance in combination with invariant learning. Decorr mitigates the issue of spurious correlations, aids in identifying stable predictors, and broadens the applicability of invariant learning methods.

Decorr: Environment Partitioning for Invariant Learning and OOD Generalization

TL;DR

Decorr tackles the challenge of improving invariant learning when natural environment labels are unavailable by algorithmically partitioning data into multiple low-correlation environments. It optimizes sample weights to minimize the weighted correlation distance to the identity, then applies IRM across the resulting environments to induce invariant predictors that generalize under covariate shifts and spurious correlations. Across synthetic, tabular, and image datasets, Decorr+IRM (and Decorr+REx) consistently outperforms ERM and other partitioning strategies, especially under model misspecification or distributional shifts. This approach broadens the applicability of invariant learning by eliminating reliance on predefined environments and offers practical gains in OOD robustness for diverse applications.

Abstract

Invariant learning methods, aimed at identifying a consistent predictor across multiple environments, are gaining prominence in out-of-distribution (OOD) generalization. Yet, when environments aren't inherent in the data, practitioners must define them manually. This environment partitioning--algorithmically segmenting the training dataset into environments--crucially affects invariant learning's efficacy but remains underdiscussed. Proper environment partitioning could broaden the applicability of invariant learning and enhance its performance. In this paper, we suggest partitioning the dataset into several environments by isolating low-correlation data subsets. Through experiments with synthetic and real data, our Decorr method demonstrates superior performance in combination with invariant learning. Decorr mitigates the issue of spurious correlations, aids in identifying stable predictors, and broadens the applicability of invariant learning methods.
Paper Structure (24 sections, 8 equations, 9 figures, 8 tables, 1 algorithm)

This paper contains 24 sections, 8 equations, 9 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Invariant Risk Minimization
  4. Environment Partitioning Methods
  5. Some Other Related Works
  6. Feature Decorrelation
  7. Covariate Shift
  8. The Proposed Method
  9. OOD Generalization under Model Misspecification
  10. Experiments
  11. Synthetic Data Experiments
  12. A Toy Example
  13. Risks of IRM
  14. Real Data Experiments
  15. Implementation Details
  16. ...and 9 more sections

Figures (9)

  • Figure 2: The example of Risks of IRM.
  • Figure 3: The Adult dataset: using race as the bias feature.
  • Figure 4: The Adult dataset: using sex as the bias feature.
  • Figure 5: CMNIST training set examples: most of 0-4 are green, and most of 5-9 are red.
  • Figure 6: CMNIST testing set examples: most of 0-4 are red, and most of 5-9 are green.
  • ...and 4 more figures