Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SBAMDT: Bayesian Additive Decision Trees with Adaptive Soft Semi-multivariate Split Rules

Stamatina Lamprinakou, Huiyan Sang, Bledar A. Konomi, Ligang Lu

TL;DR

SBAMDT addresses axis-aligned limitations of standard BART in domains with structured and unstructured features by introducing additive ensembles of hard-soft semi-multivariate decision trees. It combines knot-based multivariate splits with probabilistic routing at internal nodes, enabling adaptive smoothness and geometry-respecting partitions across the input space. The framework includes principled priors, a backfitting MCMC inference scheme with GROW/PRUNE/CHANGE moves, and a GP perspective that links the additive tree prior to a coherent stochastic process. Empirically, SBAMDT and its variants outperform BART, SBART, and BAMDT on synthetic benchmarks and NYC education data, delivering superior predictive accuracy and interpretable spatial patterns.

Abstract

Bayesian Additive Regression Trees [BART, Chipman et al., 2010] have gained significant popularity due to their remarkable predictive performance and ability to quantify uncertainty. However, standard decision tree models rely on recursive data splits at each decision node, using deterministic decision rules based on a single univariate feature. This approach limits their ability to effectively capture complex decision boundaries, particularly in scenarios involving multiple features, such as spatial domains, or when transitions are either sharp or smoothly varying. In this paper, we introduce a novel probabilistic additive decision tree model that employs a soft split rule. This method enables highly flexible splits that leverage both univariate and multivariate features, while also respecting the geometric properties of the feature domain. Notably, the probabilistic split rule adapts dynamically across decision nodes, allowing the model to account for varying levels of smoothness in the regression function. We demonstrate the utility of the proposed model through comparisons with existing tree-based models on synthetic datasets and a New York City education dataset.

SBAMDT: Bayesian Additive Decision Trees with Adaptive Soft Semi-multivariate Split Rules

TL;DR

SBAMDT addresses axis-aligned limitations of standard BART in domains with structured and unstructured features by introducing additive ensembles of hard-soft semi-multivariate decision trees. It combines knot-based multivariate splits with probabilistic routing at internal nodes, enabling adaptive smoothness and geometry-respecting partitions across the input space. The framework includes principled priors, a backfitting MCMC inference scheme with GROW/PRUNE/CHANGE moves, and a GP perspective that links the additive tree prior to a coherent stochastic process. Empirically, SBAMDT and its variants outperform BART, SBART, and BAMDT on synthetic benchmarks and NYC education data, delivering superior predictive accuracy and interpretable spatial patterns.

Abstract

Bayesian Additive Regression Trees [BART, Chipman et al., 2010] have gained significant popularity due to their remarkable predictive performance and ability to quantify uncertainty. However, standard decision tree models rely on recursive data splits at each decision node, using deterministic decision rules based on a single univariate feature. This approach limits their ability to effectively capture complex decision boundaries, particularly in scenarios involving multiple features, such as spatial domains, or when transitions are either sharp or smoothly varying. In this paper, we introduce a novel probabilistic additive decision tree model that employs a soft split rule. This method enables highly flexible splits that leverage both univariate and multivariate features, while also respecting the geometric properties of the feature domain. Notably, the probabilistic split rule adapts dynamically across decision nodes, allowing the model to account for varying levels of smoothness in the regression function. We demonstrate the utility of the proposed model through comparisons with existing tree-based models on synthetic datasets and a New York City education dataset.
Paper Structure (35 sections, 3 theorems, 32 equations, 17 figures, 4 tables, 3 algorithms)

This paper contains 35 sections, 3 theorems, 32 equations, 17 figures, 4 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Review of Semi-Multivariate Decision Tree
  3. Adaptive Soft Semi-Multivariate Additive Decision Trees
  4. Hard semi-multivariate decision rule model of reference points
  5. An adaptive Soft semi-multivariate decision rule model
  6. Priors of other model parameters.
  7. Prior on the probabilistic decisions at internal nodes.
  8. Prior on the softness control parameter.
  9. Prior on the leaf weight parameters
  10. Prior on the residual variance
  11. Prior on the decision trees.
  12. Connection between GP and the SBAMDT model
  13. Inference algorithm
  14. Simulation study on synthetic data
  15. U-shape Example
  16. ...and 20 more sections

Key Result

Theorem 3.1

Prior distribution: Posterior Distribution:

Figures (17)

  • Figure 1: (a) An example of a semi-multivariate decision tree; (b) A bipartition of the spanning tree graph $G_T^*$ into two disjoint reference point sets represented by red and blue colors, respectively.
  • Figure 2: Comparison of a hard decision tree to a soft decision tree as derived by Linero (SBART) and the hard-soft decision trees of Sk-BAMDT and S2-BAMDT. We have used a logistic function with a bandwidth parameter equal to 0.08 for SBART, $\{\alpha_1,\alpha_2\}\times q=\{0.5,1\}\times 12$ for Sk-BAMDT, and $\alpha^{(h)}\times q= 0.5\times 12$ for S2-BAMDT.
  • Figure 3: The ground truth for $f(\mathbf{s},\mathbf{x})$ and the predictive surfaces $\hat{f}(\mathbf{s},\mathbf{x})$ of each method for a U-shape simulated data. The red circle indicates discontinuity boundaries in the true function projected to the 2-D U-shape domain.
  • Figure 4: The APE of each method for one U-shape simulation.
  • Figure 5: The CRPS of SBAMDT, BAMDT, BART and SBART associated with U-shape Example.
  • ...and 12 more figures

Theorems & Definitions (4)

  • Theorem 3.1: Connection between SBAMDT and GP
  • proof
  • Proposition 3.2
  • Theorem 10.1