Optimal or Greedy Decision Trees? Revisiting their Objectives, Tuning, and Performance
Jacobus G. M. van der Linden, Daniël Vos, Mathijs M. de Weerdt, Sicco Verwer, Emir Demirović
TL;DR
This study clarifies when optimal decision trees surpass greedy approaches by dissecting training objectives and tuning. By evaluating 11 objectives and six tuning methods across 180 real and synthetic datasets, it shows that non-concave, regularized objectives can improve ODT performance, while greedy methods benefit from strictly concave criteria. Tunability is shown to be crucial for ODTs, with depth-tuning offering a fast, effective option, and the authors provide practical recommendations and open-source code to enable fair benchmarking. The findings also reveal nuanced, data-dependent differences between ODTs and greedy trees, especially regarding interpretability and scalability, and advocate for standardized evaluation practices. Collectively, the work advances understanding of ODT design, offers actionable guidance for practitioners, and lays groundwork for future, more scalable optimal-tree methods.
Abstract
Recently there has been a surge of interest in optimal decision tree (ODT) methods that globally optimize accuracy directly, in contrast to traditional approaches that locally optimize an impurity or information metric. However, the value of optimal methods is not well understood yet, as the literature provides conflicting results, with some demonstrating superior out-of-sample performance of ODTs over greedy approaches, while others show the opposite. Through a novel extensive experimental study, we provide new insights into the design and behavior of learning decision trees. In particular, we identify and analyze two relatively unexplored aspects of ODTs: the objective function used in training trees, and tuning techniques. Thus, we address these three questions: what objective to optimize in ODTs; how to tune ODTs; and how do optimal and greedy methods compare? Our experimental evaluation examines 11 objective functions, six tuning methods, and six claims from the literature on optimal and greedy methods on 180 real and synthetic data sets. Through our analysis, both conceptually and experimentally, we show the effect of (non-)concave objectives in greedy and optimal approaches; we highlight the importance of proper tuning of ODTs; support and refute several claims from the literature; provide clear recommendations for researchers and practitioners on the usage of greedy and optimal methods; and code for future comparisons.