Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Soft regression trees: a model variant and a decomposition training algorithm

Antonio Consolo, Edoardo Amaldi, Andrea Manno

TL;DR

The paper tackles the challenge of learning globally optimized decision trees by introducing Soft Regression Trees (SRTs) that predict from a single leaf along a Highest Branch Probability path, enabling conditional computation and faster inference.A convergent node-based decomposition scheme (NODEC-GS) is developed to train SRTs, with a practical variant (NODEC-DR) that adds clustering-based initialization and a data-point reassignment heuristic to balance routing and improve convergence.The authors prove a universal approximation property for SRTs, showing the leaf-only function class is dense in the space of continuous functions on any compact domain, and demonstrate superior empirical performance on 15 datasets compared to soft (ORRT) and some deterministic (ORT-L) baselines, while achieving substantial training-time reductions; Random Forest is used as a reference for interpretability vs. accuracy.Overall, the work advances interpretable, efficiently trainable soft-tree models with strong approximation capabilities and practical training schemes suitable for larger-scale problems.

Abstract

Decision trees are widely used for classification and regression tasks in a variety of application fields due to their interpretability and good accuracy. During the past decade, growing attention has been devoted to globally optimized decision trees with deterministic or soft splitting rules at branch nodes, which are trained by optimizing the error function over all the tree parameters. In this work, we propose a new variant of soft multivariate regression trees (SRTs) where, for every input vector, the prediction is defined as the linear regression associated to a single leaf node, namely, the leaf node obtained by routing the input vector from the root along the branches with higher probability. SRTs exhibit the conditional computational property, i.e., each prediction depends on a small number of nodes (parameters), and our nonlinear optimization formulation for training them is amenable to decomposition. After showing a universal approximation result for SRTs, we present a decomposition training algorithm including a clustering-based initialization procedure and a heuristic for reassigning the input vectors along the tree. Under mild assumptions, we establish asymptotic convergence guarantees. Experiments on 15 wellknown datasets indicate that our SRTs and decomposition algorithm yield higher accuracy and robustness compared with traditional soft regression trees trained using the nonlinear optimization formulation of Blanquero et al., and a significant reduction in training times as well as a slightly better average accuracy compared with the mixed-integer optimization approach of Bertsimas and Dunn. We also report a comparison with the Random Forest ensemble method.

Soft regression trees: a model variant and a decomposition training algorithm

TL;DR

The paper tackles the challenge of learning globally optimized decision trees by introducing Soft Regression Trees (SRTs) that predict from a single leaf along a Highest Branch Probability path, enabling conditional computation and faster inference.A convergent node-based decomposition scheme (NODEC-GS) is developed to train SRTs, with a practical variant (NODEC-DR) that adds clustering-based initialization and a data-point reassignment heuristic to balance routing and improve convergence.The authors prove a universal approximation property for SRTs, showing the leaf-only function class is dense in the space of continuous functions on any compact domain, and demonstrate superior empirical performance on 15 datasets compared to soft (ORRT) and some deterministic (ORT-L) baselines, while achieving substantial training-time reductions; Random Forest is used as a reference for interpretability vs. accuracy.Overall, the work advances interpretable, efficiently trainable soft-tree models with strong approximation capabilities and practical training schemes suitable for larger-scale problems.

Abstract

Decision trees are widely used for classification and regression tasks in a variety of application fields due to their interpretability and good accuracy. During the past decade, growing attention has been devoted to globally optimized decision trees with deterministic or soft splitting rules at branch nodes, which are trained by optimizing the error function over all the tree parameters. In this work, we propose a new variant of soft multivariate regression trees (SRTs) where, for every input vector, the prediction is defined as the linear regression associated to a single leaf node, namely, the leaf node obtained by routing the input vector from the root along the branches with higher probability. SRTs exhibit the conditional computational property, i.e., each prediction depends on a small number of nodes (parameters), and our nonlinear optimization formulation for training them is amenable to decomposition. After showing a universal approximation result for SRTs, we present a decomposition training algorithm including a clustering-based initialization procedure and a heuristic for reassigning the input vectors along the tree. Under mild assumptions, we establish asymptotic convergence guarantees. Experiments on 15 wellknown datasets indicate that our SRTs and decomposition algorithm yield higher accuracy and robustness compared with traditional soft regression trees trained using the nonlinear optimization formulation of Blanquero et al., and a significant reduction in training times as well as a slightly better average accuracy compared with the mixed-integer optimization approach of Bertsimas and Dunn. We also report a comparison with the Random Forest ensemble method.
Paper Structure (27 sections, 4 theorems, 74 equations, 7 figures, 9 tables, 2 algorithms)

This paper contains 27 sections, 4 theorems, 74 equations, 7 figures, 9 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Previous work
  3. Soft regression trees with single leaf node predictions
  4. The soft regression tree model variant and the training formulation
  5. Universal approximation property
  6. A general decomposition scheme
  7. Node-based decomposition scheme
  8. Asymptotic convergence
  9. Practical version of the node-based decomposition algorithm
  10. Preliminary discussion on the BN Step
  11. NODEC-DR and its asymptotic convergence
  12. BN Step optimization with data points reassignment heuristic
  13. Initialization procedure
  14. Experimental results
  15. Datasets and experimental setting
  16. ...and 12 more sections

Key Result

Theorem 1

Assuming that $\mathbb{X} \subset \mathbb{R}^p$ is an arbitrary compact set, the class $\mathbb{H}$ is dense in $\mathbb{C}(\mathbb{X})$. In other words, for any $\varepsilon > 0$ and any $g \in \mathbb{C}(\mathbb{X})$, there exists a function $f \in \mathbb{H}$ such that Since $\mathbb{H} \subset \mathbb{M}$, also the class $\mathbb{M}$ is dense in $\mathbb{C}(\mathbb{X})$.

Figures (7)

  • Figure 1: (a) Depicts an example of soft regression tree of depth $D=2$. (b) Represents the logistic CDF which corresponds to the probability of taking the left branch.
  • Figure 2: Example of SRT with single leaf node predictions. For input vector $\mathbf{x}$, the arrows indicate the branches belonging to its HBP path, while the prediction is equal to $\mathbf{\beta}_6^T \mathbf{x}$.
  • Figure 3: Example of NODEC-DR working set selection for an SRT of depth $D=3$. The branch node $t=3$ and the corresponding working sets $W_B=\{3,6,7\}$ (in red) and $W_L = \{12,13,14,15\}$ (in green) are selected. The associated variable vectors are indicated inside each node.
  • Figure 4: Example of 2 SRTs of depth $D_1=2$ (green) and, respectively, $D_2 = 1$ (red), with an input vector $\mathbf{x}$ routed along the two trees.
  • Figure 5: The new SRT of depth $D = 3$ generated by stacking the green and the red SRTs. Each leaf node is uniquely identified by a pair of numbers in which the first term represents the index belonging to the leaf of the SRT of depth $D_1$ and the second term the index of the SRT of depth $D_2$.
  • ...and 2 more figures

Theorems & Definitions (9)

  • Theorem 1
  • Proposition 1
  • Proposition 3
  • Theorem 2
  • Claim 1
  • Claim 2
  • Claim 3
  • Claim 4
  • Claim 5