Improving Random Forests by Smoothing

TL;DR

This work presents a post-learning kernel smoothing framework for piecewise-constant predictors to fuse the local adaptivity of trees with smooth uncertainty modeling. By convolving a learned predictor with a kernel and calibrating its output, the method yields differentiable predictions with an explicit uncertainty term that accounts for leaf-boundary variability; extended to random forests, the approach provides smoothed per-tree predictions averaged across the ensemble and a composite uncertainty estimate combining intra- and inter-model variance with noise. Parameter selection via out-of-bag risk enables data-driven tuning of the smoothing bandwidth and calibration. Empirical results on 14 UCI datasets show that the smoothed random forest (SRF), particularly with local smoothing, improves mean-squared error and often log-loss compared with standard RF and GP baselines, while maintaining computational efficiency. Overall, the method offers a practical and scalable approach to robust small-data regression with principled uncertainty quantification.

Abstract

Gaussian process regression is a popular model in the small data regime due to its sound uncertainty quantification and the exploitation of the smoothness of the regression function that is encountered in a wide range of practical problems. However, Gaussian processes perform sub-optimally when the degree of smoothness is non-homogeneous across the input domain. Random forest regression partially addresses this issue by providing local basis functions of variable support set sizes that are chosen in a data-driven way. However, they do so at the expense of forgoing any degree of smoothness, which often results in poor performance in the small data regime. Here, we aim to combine the advantages of both models by applying a kernel-based smoothing mechanism to a learned random forest or any other piecewise constant prediction function. As we demonstrate empirically, the resulting model consistently improves the predictive performance of the underlying random forests and, in almost all test cases, also improves the log loss of the usual uncertainty quantification based on inter-tree variance. The latter advantage can be attributed to the ability of the smoothing model to take into account the uncertainty over the exact tree-splitting locations.

Figures (5)

• Figure 1: This figure illustrates the implementation of the proposed smoothing procedures. The left panel displays the predictions from a decision tree regressor without any smoothing, using training observations represented by black points. The central panel presents the predictions after applying a Gaussian kernel for smoothing. The right panel details the probabilities assigned to each leaf of the decision tree alongside the kernel distribution for a specific query point, indicated by a red cross.
• Figure 2: Percentage improvement of MSE over the RF(100) base model for the Facebook and Fertility data sets for varying training sizes.
• Figure 3: Percentage improvement of MSE and Log-loss over the RF(100) base model for all data sets for varying training sizes.
• Figure 4: This figure shows $\text{PI}_{\text{MSE}}$ and $\text{PI}_{\text{log-loss}}$ across diverse datasets. Its data name identifies each in the subtitle. $\mathit{N}$ represents the total number of observations within each dataset, and $\mathit{p}$ indicates the number of input features. All lines are benchmarked against RF(100).
• Figure 5: The proportion of experiments that meet or exceed specific thresholds in terms of $\text{PI}_{\text{MSE}}$ (left) and $\text{PI}_{\text{log-loss}}$ (right), indicating improvements in model performance compared to the baseline, which is RF 100 and GP.

Theorems & Definitions (1)

• Definition 1