Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Covariance-Driven Regression Trees: Reducing Overfitting in CART

Likun Zhang, Wei Ma

TL;DR

CovRT addresses overfitting in regression trees by replacing CART's empirical risk criterion with a covariance-driven splitting score $CS(j,s,t)$ and its empirical version $\widehat{CS}(j,s,t)$. This criterion favors splits on covariates with true signal while imposing a balance penalty through $\widehat{P}_{t_L}\widehat{P}_{t_R}$, reducing end-cut bias. The authors prove an oracle inequality and high-dimensional consistency, showing predictive accuracy comparable to CART and often superior in finite samples, as demonstrated in simulations and real datasets such as Boston Housing, Airfoil Self-Noise, and Abalone. The work suggests CovRT as a robust component for tree-based methods and a pathway to extensions to classification, ensembles, and causal inference.

Abstract

Decision trees are powerful machine learning algorithms, widely used in fields such as economics and medicine for their simplicity and interpretability. However, decision trees such as CART are prone to overfitting, especially when grown deep or the sample size is small. Conventional methods to reduce overfitting include pre-pruning and post-pruning, which constrain the growth of uninformative branches. In this paper, we propose a complementary approach by introducing a covariance-driven splitting criterion for regression trees (CovRT). This method is more robust to overfitting than the empirical risk minimization criterion used in CART, as it produces more balanced and stable splits and more effectively identifies covariates with true signals. We establish an oracle inequality of CovRT and prove that its predictive accuracy is comparable to that of CART in high-dimensional settings. We find that CovRT achieves superior prediction accuracy compared to CART in both simulations and real-world tasks.

Covariance-Driven Regression Trees: Reducing Overfitting in CART

TL;DR

CovRT addresses overfitting in regression trees by replacing CART's empirical risk criterion with a covariance-driven splitting score and its empirical version . This criterion favors splits on covariates with true signal while imposing a balance penalty through , reducing end-cut bias. The authors prove an oracle inequality and high-dimensional consistency, showing predictive accuracy comparable to CART and often superior in finite samples, as demonstrated in simulations and real datasets such as Boston Housing, Airfoil Self-Noise, and Abalone. The work suggests CovRT as a robust component for tree-based methods and a pathway to extensions to classification, ensembles, and causal inference.

Abstract

Decision trees are powerful machine learning algorithms, widely used in fields such as economics and medicine for their simplicity and interpretability. However, decision trees such as CART are prone to overfitting, especially when grown deep or the sample size is small. Conventional methods to reduce overfitting include pre-pruning and post-pruning, which constrain the growth of uninformative branches. In this paper, we propose a complementary approach by introducing a covariance-driven splitting criterion for regression trees (CovRT). This method is more robust to overfitting than the empirical risk minimization criterion used in CART, as it produces more balanced and stable splits and more effectively identifies covariates with true signals. We establish an oracle inequality of CovRT and prove that its predictive accuracy is comparable to that of CART in high-dimensional settings. We find that CovRT achieves superior prediction accuracy compared to CART in both simulations and real-world tasks.
Paper Structure (18 sections, 7 theorems, 99 equations, 4 figures, 2 tables, 1 algorithm)

This paper contains 18 sections, 7 theorems, 99 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Framework and Notation
  3. Regression Tree Construction
  4. CovRT Methodology
  5. Overfitting in CART
  6. Statistical Properties of CovRT
  7. Oracle Inequality and Consistency
  8. Robustness to Overfitting
  9. Simulation Study
  10. Empirical Applications
  11. Discussion
  12. Additional theoretical results
  13. Proof of the main results
  14. Proof of Theorem 1
  15. Proof of Theorem 2
  16. ...and 3 more sections

Key Result

Theorem 1

Suppose that $g^*(\mathbf{x}) \in \mathcal{G}^1$ and that the coordinates $x_1, \ldots, x_p$ are mutually independent, each with marginal distribution $F_{\textbf{x}_j}$. Let $F_{\textbf{x}_j\mid \mathrm{t}}$ denote the distribution of $x_j$ conditional on $\mathbf{x} \in \mathrm{t}$. Then at any no

Figures (4)

  • Figure 1: Depth-1 (orange solid line) and depth-2 (green solid line) trees split by $\mathcal{CS}(j, s, \mathrm{t})$ with $g^*(x) = x^3$ (black line) and $x$ uniformly distributed on $(-1, 1]$. The dashed lines show the split points.
  • Figure 2: Empirical $L_2$ risk for CART (red) and CovRT (blue) with the number of leaves ranging from 1 to 20, evaluated on the training data (dashed lines) and the testing data (solid lines). Points indicate averages over 500 replications.
  • Figure 3: Density for split points $\hat{s}$ for purely random splitting tree (green), CART (red) and CovRT (blue) where $g^* = c_0 + c_1x_{1}$ for $c_0 = 1, c_1 = 0$ (left: noisy), $c_0 = 1, c_1 = 0.5$ (middle: medium signal) and $c_0 = 1, c_1 = 1$ (right: strong signal).
  • Figure 4: Accuracy for purely random splitting tree (green), CART (red), and CovRT (blue) in correctly splitting on the true signal covariate $x_1$.

Theorems & Definitions (14)

  • Theorem 1
  • Example 1: Multivariate linear model
  • Example 2: Monotone component functions
  • Theorem 2
  • Lemma 1: Excess empirical risk within node
  • Theorem 3: Empirical risk bound
  • Theorem 4: Oracle inequality
  • Corollary 1: Consistency
  • Proposition 1
  • proof
  • ...and 4 more