Learning accurate and interpretable tree-based models

TL;DR

This work advances data-driven design of tree-based models by treating hyperparameter selection (splitting criteria, priors, pruning, and ensembles) as a learning problem across multiple related datasets from the same domain. It introduces a flexible $(oldsymbol{ ho})$-Tsallis entropy family for node splitting, proves sample-complexity bounds for tuning these parameters (and for related priors and pruning parameters), and develops efficient, output-sensitive algorithms to implement ERM in practice. The authors extend the theory to regression trees, random forests, and gradient-boosted trees, and they formulate objectives that balance explainability with accuracy. Empirical results on UCI datasets corroborate that data-driven tuning yields data-specific gains and reveals substantial variability in optimal hyperparameters across domains, underscoring the value of adaptive, principled tree design for interpretability and performance.

Abstract

Decision trees and their ensembles are popular in machine learning as easy-to-understand models. Several techniques have been proposed in the literature for learning tree-based classifiers, with different techniques working well for data from different domains. In this work, we develop approaches to design tree-based learning algorithms given repeated access to data from the same domain. We study multiple formulations covering different aspects and popular techniques for learning decision tree based approaches. We propose novel parameterized classes of node splitting criteria in top-down algorithms, which interpolate between popularly used entropy and Gini impurity based criteria, and provide theoretical bounds on the number of samples needed to learn the splitting function appropriate for the data at hand. We also study the sample complexity of tuning prior parameters in Bayesian decision tree learning, and extend our results to decision tree regression. We further consider the problem of tuning hyperparameters in pruning the decision tree for classical pruning algorithms including min-cost complexity pruning. In addition, our techniques can be used to optimize the explainability versus accuracy trade-off when using decision trees. We extend our results to tuning popular tree-based ensembles, including random forests and gradient-boosted trees. We demonstrate the significance of our approach on real world datasets by learning data-specific decision trees which are simultaneously more accurate and interpretable.

Figures (6)

• Figure 1: The loss of pruned tree as a function of the mininum cost-complexity pruning parameter $\tilde{\alpha}$ is piecewise constant with at most $t$ pieces. The optimal complexity parameter $\tilde{\alpha}$ varies with the dataset.
• Figure 2: Accuracy vs $\eta*|\mathrm{leaves}(T)|$ as the pruning parameter $\tilde{\alpha}$ is varied, for $\eta=0.01$.
• Figure 3: Average test accuracy (proportional to brightness, yellow is highest) of $(\alpha,\beta)$-Tsallis entropy based splitting criterion as the parameters are varied, across datasets. We observe that different parameter settings work best for each dataset, highlighting the need to learn data-specific values.
• Figure 4: Average test accuracy as a function of the regularization strength $\lambda$ in XGBoost, across datasets. Different parameter settings work best for different datasets.
• Figure 5: Accuracy-explainability frontier for different $\alpha$ or different $\beta$ in the $(\alpha,\beta)$-Tsallis entropy family for the splitting criterion, as the pruning parameter $\tilde{\alpha}$ is varied.

Theorems & Definitions (34)

• Definition 1: Shattering and Pseudo-dimension, anthony1999neural
• Theorem 2.1: Uniform convergence sample complexity via pseudo-dimension, anthony1999neural
• Lemma 2.2
• Proposition 3.1
• proof
• Proposition 3.2
• Theorem 3.3
• proof
• Theorem 3.4
• proof