Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exogenous Randomness Empowering Random Forests

Tianxing Mei, Yingying Fan, Jinchi Lv

TL;DR

The results reveal an intriguing phenomenon: the presence of noise features can act as a "blessing" in enhancing the performance of random forests thanks to feature subsampling.

Abstract

We offer theoretical and empirical insights into the impact of exogenous randomness on the effectiveness of random forests with tree-building rules independent of training data. We formally introduce the concept of exogenous randomness and identify two types of commonly existing randomness: Type I from feature subsampling, and Type II from tie-breaking in tree-building processes. We develop non-asymptotic expansions for the mean squared error (MSE) for both individual trees and forests and establish sufficient and necessary conditions for their consistency. In the special example of the linear regression model with independent features, our MSE expansions are more explicit, providing more understanding of the random forests' mechanisms. It also allows us to derive an upper bound on the MSE with explicit consistency rates for trees and forests. Guided by our theoretical findings, we conduct simulations to further explore how exogenous randomness enhances random forest performance. Our findings unveil that feature subsampling reduces both the bias and variance of random forests compared to individual trees, serving as an adaptive mechanism to balance bias and variance. Furthermore, our results reveal an intriguing phenomenon: the presence of noise features can act as a "blessing" in enhancing the performance of random forests thanks to feature subsampling.

Exogenous Randomness Empowering Random Forests

TL;DR

The results reveal an intriguing phenomenon: the presence of noise features can act as a "blessing" in enhancing the performance of random forests thanks to feature subsampling.

Abstract

We offer theoretical and empirical insights into the impact of exogenous randomness on the effectiveness of random forests with tree-building rules independent of training data. We formally introduce the concept of exogenous randomness and identify two types of commonly existing randomness: Type I from feature subsampling, and Type II from tie-breaking in tree-building processes. We develop non-asymptotic expansions for the mean squared error (MSE) for both individual trees and forests and establish sufficient and necessary conditions for their consistency. In the special example of the linear regression model with independent features, our MSE expansions are more explicit, providing more understanding of the random forests' mechanisms. It also allows us to derive an upper bound on the MSE with explicit consistency rates for trees and forests. Guided by our theoretical findings, we conduct simulations to further explore how exogenous randomness enhances random forest performance. Our findings unveil that feature subsampling reduces both the bias and variance of random forests compared to individual trees, serving as an adaptive mechanism to balance bias and variance. Furthermore, our results reveal an intriguing phenomenon: the presence of noise features can act as a "blessing" in enhancing the performance of random forests thanks to feature subsampling.

Paper Structure

This paper contains 38 sections, 14 theorems, 299 equations, 10 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related literature
  3. Ensemble estimator with exogenous randomness
  4. Partitioning estimator
  5. Ensemble estimator with exogenous randomness
  6. Advantages of ensemble in CART random forests
  7. Theoretical analysis
  8. Model with binary features
  9. Model with continuous features
  10. Additional insights via theory-guided simulations
  11. Random forests can reduce both squared bias and variance
  12. Random forests can outperform decision trees even with $\gamma=1$
  13. Feature subsampling improves the performance of random forests
  14. General results on partitioning estimator ensemble
  15. Local MSE bounds for ensemble estimators
  16. ...and 23 more sections

Key Result

Theorem 1

Let $I_l$ and $I'_{l}$ be two independent Binary CART processes. The global MSE of the random forests estimator is given by where the remainder satisfies $\mathcal{R}_{RF} \lesssim \frac{2^l}{n(1 + (n-1) 2^{-l})^{1/2}} + \left(1 - 2^{-l} \right)^n.$ In particular, when $B =1$, the global MSE of a single tree estimator is in which the remainder is bounded by $\mathcal{R}_{\text{tree}} \lesssim \f

Figures (10)

  • Figure 1: Performance measures (a)--(f) for tree and forests in the binary case under configuration (I) as $\gamma$ varies. Tree depth is fixed at $l = 7$. Red: tree; blue: forest.
  • Figure 2: Performance measures (a)--(f) for tree and forest in the continuous case under configuration (I) as $\gamma$ varies. Tree depth is fixed at $l = 7$. Red: tree; blue: forest.
  • Figure 3: Performance measures (a)--(f) for tree and forest in the binary case under configuration (I) as $l$ varies. We fix $\gamma = 1$. Red: tree; blue: forest.
  • Figure 4: Performance measures (a)--(f) for tree and forest in the continuous case under configuration (I) as $l$ varies. We fix $\gamma = 1$. Red: tree; blue: forest.
  • Figure 5: Performance measures (a)--(f) for forests in the binary case as $\gamma$ and $l$ vary under configuration (I).
  • ...and 5 more figures

Theorems & Definitions (23)

  • Definition 1: Exogenous randomness in population CART
  • Definition 2: Binary CART process
  • Theorem 1
  • Remark 1
  • Remark 2
  • Remark 3
  • Definition 3: Uniform CART process
  • Theorem 2
  • Remark 4
  • Remark 5
  • ...and 13 more