Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Theoretical and Empirical Advances in Forest Pruning

Albert Dorador

TL;DR

The paper addresses the challenge of achieving high predictive accuracy from regression forests while preserving interpretability by pruning ensembles. It introduces and analyzes four pruning approaches—SFS, SBS', BSF, and non-negative Lasso—proving an asymptotic advantage for Lasso pruning and providing finite-sample generalization bounds. Through extensive simulations and real-data experiments across 19 datasets, the authors show that pruned forests often match or exceed the full forest's accuracy while using only a small fraction of trees, with particular gains in larger-sample, higher-signal settings. It also demonstrates a practical path to interpretability by merging a small sub-forest into a single, more transparent tree without sacrificing performance in many cases.

Abstract

Regression forests have long delivered state-of-the-art accuracy, often outperforming regression trees and even neural networks, but they suffer from limited interpretability as ensemble methods. In this work, we revisit forest pruning, an approach that aims to have the best of both worlds: the accuracy of regression forests and the interpretability of regression trees. This pursuit, whose foundation lies at the core of random forest theory, has seen vast success in empirical studies. In this paper, we contribute theoretical results that support and qualify those empirical findings; namely, we prove the asymptotic advantage of a Lasso-pruned forest over its unpruned counterpart under weak assumptions, as well as high-probability finite-sample generalization bounds for regression forests pruned according to the main methods, which we then validate by way of simulation. Then, we test the accuracy of pruned regression forests against their unpruned counterparts on 19 different datasets (16 synthetic, 3 real). We find that in the vast majority of scenarios tested, there is at least one forest-pruning method that yields equal or better accuracy than the original full forest (in expectation), while just using a small fraction of the trees. We show that, in some cases, the reduction in the size of the forest is so dramatic that the resulting sub-forest can be meaningfully merged into a single tree, obtaining a level of interpretability that is qualitatively superior to that of the original regression forest, which remains a black box.

Theoretical and Empirical Advances in Forest Pruning

TL;DR

The paper addresses the challenge of achieving high predictive accuracy from regression forests while preserving interpretability by pruning ensembles. It introduces and analyzes four pruning approaches—SFS, SBS', BSF, and non-negative Lasso—proving an asymptotic advantage for Lasso pruning and providing finite-sample generalization bounds. Through extensive simulations and real-data experiments across 19 datasets, the authors show that pruned forests often match or exceed the full forest's accuracy while using only a small fraction of trees, with particular gains in larger-sample, higher-signal settings. It also demonstrates a practical path to interpretability by merging a small sub-forest into a single, more transparent tree without sacrificing performance in many cases.

Abstract

Regression forests have long delivered state-of-the-art accuracy, often outperforming regression trees and even neural networks, but they suffer from limited interpretability as ensemble methods. In this work, we revisit forest pruning, an approach that aims to have the best of both worlds: the accuracy of regression forests and the interpretability of regression trees. This pursuit, whose foundation lies at the core of random forest theory, has seen vast success in empirical studies. In this paper, we contribute theoretical results that support and qualify those empirical findings; namely, we prove the asymptotic advantage of a Lasso-pruned forest over its unpruned counterpart under weak assumptions, as well as high-probability finite-sample generalization bounds for regression forests pruned according to the main methods, which we then validate by way of simulation. Then, we test the accuracy of pruned regression forests against their unpruned counterparts on 19 different datasets (16 synthetic, 3 real). We find that in the vast majority of scenarios tested, there is at least one forest-pruning method that yields equal or better accuracy than the original full forest (in expectation), while just using a small fraction of the trees. We show that, in some cases, the reduction in the size of the forest is so dramatic that the resulting sub-forest can be meaningfully merged into a single tree, obtaining a level of interpretability that is qualitatively superior to that of the original regression forest, which remains a black box.
Paper Structure (43 sections, 11 theorems, 48 equations, 17 figures, 40 tables, 4 algorithms)

This paper contains 43 sections, 11 theorems, 48 equations, 17 figures, 40 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. The forest pruning problem
  3. Problem definition
  4. Sequential forward selection (SFS)
  5. Modified sequential backward selection (SBS')
  6. The best sub-forest (BSF) method
  7. The (non-negative) Lasso solution
  8. Visualizing the pruned forest
  9. Theoretical results
  10. The asymptotic advantage of Lasso pruning
  11. Generalization bound for Lasso-pruned regression forests
  12. Testing conditions common to all experiments
  13. Simulations with synthetic data
  14. Experiments with real data
  15. From sub-forest to tree
  16. ...and 28 more sections

Key Result

Theorem 4.2

(Out-of-sample risk bound of non-negative Lasso) With the previous (essentially assumption-free) setup, where $\hat{\beta}$ satisfies for some $\tau>0$. Therefore,

Figures (17)

  • Figure 1: Ensemble of 200 CART trees pruned down to two by BSF (Diamonds dataset)
  • Figure 2: MSPE of full forest (FF) and forest-pruning methods under the worst (left) and best (right) conditions
  • Figure 3: MSPE and average # trees (across 100 repetitions) in full and pruned forests in Diamonds
  • Figure 4: MSPE of the full forest, pruned tree and BSF-pruned forest in the Diamonds dataset
  • Figure 5: MSPE of full and pruned forests in scenarios with 2 relevant variables and low noise
  • ...and 12 more figures

Theorems & Definitions (36)

  • Definition 4.1
  • Theorem 4.2
  • proof
  • Corollary 4.3
  • proof
  • Remark 4.4
  • Theorem 4.5
  • proof
  • Theorem 4.6
  • proof
  • ...and 26 more