Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Consistency of Honest Decision Trees and Random Forests

Martin Bladt, Rasmus Frigaard Lemvig

TL;DR

This work provides an elementary, kernel-like analysis of the consistency of honest decision trees and honest random forests in regression. It establishes weak, strong, and uniform consistency under mild regularity, and extends to a flexible double-bootstrap framework that includes two-stage subsampling; several known results emerge as special cases. By recasting tree averages as adaptive kernel-type smoothers, the paper clarifies the link between data-driven partitioning and classical smoothing theory while enabling new ensemble variants. The results are framed with accessible proofs and concrete splitting schemes, offering practical guidance for ensuring consistency across different bootstrap regimes.

Abstract

We study various types of consistency of honest decision trees and random forests in the regression setting. In contrast to related literature, our proofs are elementary and follow the classical arguments used for smoothing methods. Under mild regularity conditions on the regression function and data distribution, we establish weak and almost sure convergence of honest trees and honest forest averages to the true regression function, and moreover we obtain uniform convergence over compact covariate domains. The framework naturally accommodates ensemble variants based on subsampling and also a two-stage bootstrap sampling scheme. Our treatment synthesizes and simplifies existing analyses, in particular recovering several results as special cases. The elementary nature of the arguments clarifies the close relationship between data-adaptive partitioning and kernel-type methods, providing an accessible approach to understanding the asymptotic behavior of tree-based methods.

Consistency of Honest Decision Trees and Random Forests

TL;DR

This work provides an elementary, kernel-like analysis of the consistency of honest decision trees and honest random forests in regression. It establishes weak, strong, and uniform consistency under mild regularity, and extends to a flexible double-bootstrap framework that includes two-stage subsampling; several known results emerge as special cases. By recasting tree averages as adaptive kernel-type smoothers, the paper clarifies the link between data-driven partitioning and classical smoothing theory while enabling new ensemble variants. The results are framed with accessible proofs and concrete splitting schemes, offering practical guidance for ensuring consistency across different bootstrap regimes.

Abstract

We study various types of consistency of honest decision trees and random forests in the regression setting. In contrast to related literature, our proofs are elementary and follow the classical arguments used for smoothing methods. Under mild regularity conditions on the regression function and data distribution, we establish weak and almost sure convergence of honest trees and honest forest averages to the true regression function, and moreover we obtain uniform convergence over compact covariate domains. The framework naturally accommodates ensemble variants based on subsampling and also a two-stage bootstrap sampling scheme. Our treatment synthesizes and simplifies existing analyses, in particular recovering several results as special cases. The elementary nature of the arguments clarifies the close relationship between data-adaptive partitioning and kernel-type methods, providing an accessible approach to understanding the asymptotic behavior of tree-based methods.
Paper Structure (21 sections, 37 theorems, 236 equations)

This paper contains 21 sections, 37 theorems, 236 equations.

Table of Contents

  1. Introduction
  2. Consistency for a single tree
  3. Assumptions
  4. Weak consistency of decision trees via honesty
  5. Strong consistency
  6. Uniform consistency
  7. General splitting criteria
  8. Concrete splitting criteria
  9. Uniform splitting
  10. Centered splitting
  11. Bootstrapping and random forests
  12. Bootstrap assumptions
  13. Weak consistency
  14. Strong consistency
  15. Proofs
  16. ...and 6 more sections

Key Result

Lemma 2.1

Fix $\delta > 0$. It holds that and that

Theorems & Definitions (83)

  • Lemma 2.1: Conditioning
  • Proposition 2.2: Weak consistency of the density estimator
  • Theorem 2.3: Weak consistency of decision trees
  • Theorem 2.4: $L_1$ and $L_p$ convergence of decision trees
  • Proposition 2.5: Strong consistency of the density estimator
  • Theorem 2.6: Strong consistency of decision trees
  • Theorem 2.7: Uniform consistency of decision trees
  • Corollary 2.8: Uniform consistency of decision trees with bounded responses
  • Proposition 2.9: Consistency under balance condition
  • Proposition 2.10: Consistency under minimal node size
  • ...and 73 more