Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Empowering Decision Trees via Shape Function Branching

Nakul Upadhya, Eldan Cohen

TL;DR

The paper proposes Shape Generalized Trees (SGTs), where each internal node routes data using a learnable shape function on a single feature, enabling non-linear splits within one node and improving interpretability through visualizable node shapes. ShapeCART provides an efficient top-down induction for SGTs, with extensions to bivariate shape functions (S$^2$GT) and multi-way branching (SGT$_K$), plus Shape$^2$CART and ShapeCART$_K$. The authors prove expressiveness advantages over axis-aligned linear trees and show empirical gains across 26 datasets, including compact trees and favorable visualization of decision mechanisms. They also introduce a two-stage shape-function learning procedure, a pairwise-feature heuristic to limit bivariate constructions, and post-processing via TAO for global refinement. Overall, SGTs achieve superior performance with smaller model sizes while maintaining interpretability, demonstrating practical potential for interpretable non-linear partitioning in tabular data.

Abstract

Decision trees are prized for their interpretability and strong performance on tabular data. Yet, their reliance on simple axis-aligned linear splits often forces deep, complex structures to capture non-linear feature effects, undermining human comprehension of the constructed tree. To address this limitation, we propose a novel generalization of a decision tree, the Shape Generalized Tree (SGT), in which each internal node applies a learnable axis-aligned shape function to a single feature, enabling rich, non-linear partitioning in one split. As users can easily visualize each node's shape function, SGTs are inherently interpretable and provide intuitive, visual explanations of the model's decision mechanisms. To learn SGTs from data, we propose ShapeCART, an efficient induction algorithm for SGTs. We further extend the SGT framework to bivariate shape functions (S$^2$GT) and multi-way trees (SGT$_K$), and present Shape$^2$CART and ShapeCART$_K$, extensions to ShapeCART for learning S$^2$GTs and SGT$_K$s, respectively. Experiments on various datasets show that SGTs achieve superior performance with reduced model size compared to traditional axis-aligned linear trees.

Empowering Decision Trees via Shape Function Branching

TL;DR

The paper proposes Shape Generalized Trees (SGTs), where each internal node routes data using a learnable shape function on a single feature, enabling non-linear splits within one node and improving interpretability through visualizable node shapes. ShapeCART provides an efficient top-down induction for SGTs, with extensions to bivariate shape functions (SGT) and multi-way branching (SGT), plus ShapeCART and ShapeCART. The authors prove expressiveness advantages over axis-aligned linear trees and show empirical gains across 26 datasets, including compact trees and favorable visualization of decision mechanisms. They also introduce a two-stage shape-function learning procedure, a pairwise-feature heuristic to limit bivariate constructions, and post-processing via TAO for global refinement. Overall, SGTs achieve superior performance with smaller model sizes while maintaining interpretability, demonstrating practical potential for interpretable non-linear partitioning in tabular data.

Abstract

Decision trees are prized for their interpretability and strong performance on tabular data. Yet, their reliance on simple axis-aligned linear splits often forces deep, complex structures to capture non-linear feature effects, undermining human comprehension of the constructed tree. To address this limitation, we propose a novel generalization of a decision tree, the Shape Generalized Tree (SGT), in which each internal node applies a learnable axis-aligned shape function to a single feature, enabling rich, non-linear partitioning in one split. As users can easily visualize each node's shape function, SGTs are inherently interpretable and provide intuitive, visual explanations of the model's decision mechanisms. To learn SGTs from data, we propose ShapeCART, an efficient induction algorithm for SGTs. We further extend the SGT framework to bivariate shape functions (SGT) and multi-way trees (SGT), and present ShapeCART and ShapeCART, extensions to ShapeCART for learning SGTs and SGTs, respectively. Experiments on various datasets show that SGTs achieve superior performance with reduced model size compared to traditional axis-aligned linear trees.
Paper Structure (49 sections, 5 theorems, 15 equations, 11 figures, 9 tables, 4 algorithms)

This paper contains 49 sections, 5 theorems, 15 equations, 11 figures, 9 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Decision Trees
  4. Interpretability
  5. Shape Generalized Tree
  6. Expressiveness Guarantees for SGTs
  7. ShapeCART
  8. Extending to Shape$^2$CART
  9. Learning Shape Functions
  10. Binning the samples
  11. Mapping Bins to Branches
  12. Extending to Multi-Way Branching
  13. Limiting Bivariate Shape Function Construction
  14. Post-Processing
  15. Experimental Evaluation
  16. ...and 34 more sections

Key Result

Theorem 1

Every function that can be represented by a binary axis-aligned linear tree (Definition def:baalc), can be represented by an SGT (Definition eqn:sgt_cut) with the same number of decision nodes.

Figures (11)

  • Figure 1: Demonstration of an SGT on a simple toy dataset. As seen here, the SGT can achieve perfect accuracy with a smaller depth and number of nodes compared to the linear axis-aligned tree.
  • Figure 2: Test accuracy per depth for axis-aligned approaches on eye-movements, electricity, and eye-state. Top: TDIDT-Approaches. Bottom: Non-Greedy Approaches.
  • Figure 3: Visualization of a SGT induced via ShapeCART (max depth 3) trained on the eye-movements dataset. Purple indicates regions where samples will be directed to the left child; Orange directed to the right.
  • Figure : Fit Shape Function
  • Figure G.5: Test accuracy per depth for Bivariate models on eye-movements, electricity, and eye-state. Top: TDIDT-Approaches. Bottom: Non-Greedy Approaches.
  • ...and 6 more figures

Theorems & Definitions (16)

  • Definition 1: Binary Axis-Aligned Linear Tree
  • Definition 2: Binary $M$-variate Oblique Tree
  • Definition 3: Binary Axis-Aligned Shape Generalized Tree (SGT)
  • Definition 4: Binary Bivariate Shape Generalized Tree (S$^2$GT)
  • Definition 5: Multi-Way Axis-Aligned Shape Generalized Tree (SGT$_K$)
  • Theorem 1
  • Theorem 2
  • Lemma 1
  • Definition 6: Binary Multi-Way Shape Generalized Tree (S$^2$GT$_K$)
  • proof
  • ...and 6 more