Ensembles of Probabilistic Regression Trees

TL;DR

This work develops and analyzes ensembles of probabilistic regression trees (PR trees), including PR-RF, PR-GBT, and P-BART, and proves their consistency under standard assumptions. By blending soft region memberships with ensemble methods, it demonstrates favorable bias-variance properties and competitive predictive accuracy against strong regressors such as RF, GBT, and BART across diverse datasets. Theoretical results establish consistency and posterior concentration for the Bayesian variant, while extensive experiments illustrate practical performance and trade-offs, highlighting that no single method dominates and choices depend on the bias-variance emphasis. The study advances a principled, smooth alternative to traditional tree ensembles with potential for uncertainty quantification via Bayesian extensions and quantile regression.

Abstract

Tree-based ensemble methods such as random forests, gradient-boosted trees, and Bayesianadditive regression trees have been successfully used for regression problems in many applicationsand research studies. In this paper, we study ensemble versions of probabilisticregression trees that provide smooth approximations of the objective function by assigningeach observation to each region with respect to a probability distribution. We prove thatthe ensemble versions of probabilistic regression trees considered are consistent, and experimentallystudy their bias-variance trade-off and compare them with the state-of-the-art interms of performance prediction.

Figures (4)

• Figure 1: Evolution of the performance with respect to the depth of the tree on several data sets: Bike Day (BD), Boston (BO), Diabetes (DI). Left: bias, middle: variance, right: MSE.
• Figure 2: Evolution of the performance of one tree (first row), bagging methods (second row), boosting methods (third row), and Bayesian ensemble methods (fourth row) for the Diabetes data set. Left: bias, middle: variance, right: MSE. For one tree, we increase the depth of the tree to vary the dimension, whereas for the ensemble methods, we increase the number of trees.
• Figure 3: Topology $\mathcal{T}$ of $h$
• Figure 4: Topology $\mathcal{T}_1,\mathcal{T}_2,\mathcal{T}_3,\mathcal{T}_4$ of $h_1,h_2,h_3,h_4$

Theorems & Definitions (15)

• Theorem 1
• Theorem 2
• Proof 1
• Theorem 3
• Proof 2
• Proposition 1: A posteriori distribution of $R^{(\ell),.}|T^{(\ell)},\boldsymbol{\gamma}^{(\ell)},\Sigma_K$
• Proposition 2: A posteriori distribution of $\gamma_k^{(\ell)}$
• Proposition 3
• Proposition 4
• Theorem 4