Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Invariant Random Forest: Tree-Based Model Solution for OOD Generalization

Yufan Liao, Qi Wu, Xing Yan

TL;DR

The paper addresses Out-Of-Distribution generalization for tree-based models by introducing Invariant Decision Tree (IDT) and Invariant Random Forest (IRF), which incorporate a penalty to favor splits that are stable across multiple environments. A key theoretical contribution is the invariant ratio $I(Q_m^e, \theta) = CR_{X_j \leq c}^1(Q_m^e) / CR_{X_j \leq c}^0(Q_m^e)$ that remains constant when splits use stable variables, guiding the splitting process. The authors formulate a penalized objective that combines standard impurity with an invariance penalty, enabling practical tree growth under distribution shifts, and validate the approach on synthetic and real datasets where IRF outperforms RF and XGBoost in OOD settings. The work demonstrates that incorporating environment-aware invariance into tree-based methods yields substantial improvements in robustness and generalization, with potential implications for safety-critical and time-varying domains.

Abstract

Out-Of-Distribution (OOD) generalization is an essential topic in machine learning. However, recent research is only focusing on the corresponding methods for neural networks. This paper introduces a novel and effective solution for OOD generalization of decision tree models, named Invariant Decision Tree (IDT). IDT enforces a penalty term with regard to the unstable/varying behavior of a split across different environments during the growth of the tree. Its ensemble version, the Invariant Random Forest (IRF), is constructed. Our proposed method is motivated by a theoretical result under mild conditions, and validated by numerical tests with both synthetic and real datasets. The superior performance compared to non-OOD tree models implies that considering OOD generalization for tree models is absolutely necessary and should be given more attention.

Invariant Random Forest: Tree-Based Model Solution for OOD Generalization

TL;DR

The paper addresses Out-Of-Distribution generalization for tree-based models by introducing Invariant Decision Tree (IDT) and Invariant Random Forest (IRF), which incorporate a penalty to favor splits that are stable across multiple environments. A key theoretical contribution is the invariant ratio that remains constant when splits use stable variables, guiding the splitting process. The authors formulate a penalized objective that combines standard impurity with an invariance penalty, enabling practical tree growth under distribution shifts, and validate the approach on synthetic and real datasets where IRF outperforms RF and XGBoost in OOD settings. The work demonstrates that incorporating environment-aware invariance into tree-based methods yields substantial improvements in robustness and generalization, with potential implications for safety-critical and time-varying domains.

Abstract

Out-Of-Distribution (OOD) generalization is an essential topic in machine learning. However, recent research is only focusing on the corresponding methods for neural networks. This paper introduces a novel and effective solution for OOD generalization of decision tree models, named Invariant Decision Tree (IDT). IDT enforces a penalty term with regard to the unstable/varying behavior of a split across different environments during the growth of the tree. Its ensemble version, the Invariant Random Forest (IRF), is constructed. Our proposed method is motivated by a theoretical result under mild conditions, and validated by numerical tests with both synthetic and real datasets. The superior performance compared to non-OOD tree models implies that considering OOD generalization for tree models is absolutely necessary and should be given more attention.
Paper Structure (24 sections, 1 theorem, 20 equations, 1 figure, 8 tables)

This paper contains 24 sections, 1 theorem, 20 equations, 1 figure, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Invariant Learning
  4. Stable Learning
  5. Time Robust Tree
  6. Motivation: Stable Split and Unstable Split
  7. The Theorem of An Invariant
  8. Invariant Tree and Forest
  9. Invariant Classification Tree
  10. Invariant Regression Tree
  11. Invariant Random Forest
  12. Experiments
  13. Hyperparameter Settings
  14. Synthetic Data Experiments
  15. Settings of Real Data Experiments
  16. ...and 9 more sections

Key Result

Theorem 1

For any subset $D \subset \mathbb{R}^{r+t}$, let $\tilde{P_0}, \tilde{Q_0^e}, \tilde{P_1}, \tilde{Q_1^e}$ be the distribution of $P_0,Q_0^e,P_1,Q_1^e$ restricted on $D$, respectively. For a splitting rule $S_i \leq c$ on $D$, define the changing rate of positive label as and the changing rate of negative label as These changing rates can be calculated in any environment. Then, using any stable

Figures (1)

  • Figure 1: The illustration of the example in \ref{['generalMOT']}. The number in the circle is $\mathbb{E}[Y]$. The prediction rule $X_2 = 1 \rightarrow Y = 1$ has different accuracies under two different training environments, so it is questionable if this rule can still be effective under the testing environment.

Theorems & Definitions (2)

  • Theorem 1
  • proof