Better Trees: An empirical study on hyperparameter tuning of classification decision tree induction algorithms

TL;DR

This study systematically investigates hyperparameter tuning for two widely used decision-tree inductors, CART and C4.5/J48, across 94 OpenML datasets using six tuning strategies, nested cross-validation, and a fixed budget of $900$ evaluations. CART shows significant predictive gains from tuning in about $66\%$ of datasets, while J48 gains are more modest (roughly $38\%$), with IRACE consistently delivering strong performance under realistic budgets. A functional ANOVA (fANOVA) analysis reveals that only a small subset of HPs largely drives performance (notably M and C for J48; minsplit and minbucket for CART), and a meta-learning framework demonstrates how to predict when tuning is beneficial, enabling targeted, interpretable recommendations. Together, these findings inform practical guidelines for tuning decision tree HPs, highlight the value of IRACE, and point to future work on broader algorithms and automated, dataset-aware tuning strategies.

Abstract

Machine learning algorithms often contain many hyperparameters (HPs) whose values affect the predictive performance of the induced models in intricate ways. Due to the high number of possibilities for these HP configurations and their complex interactions, it is common to use optimization techniques to find settings that lead to high predictive performance. However, insights into efficiently exploring this vast space of configurations and dealing with the trade-off between predictive and runtime performance remain challenging. Furthermore, there are cases where the default HPs fit the suitable configuration. Additionally, for many reasons, including model validation and attendance to new legislation, there is an increasing interest in interpretable models, such as those created by the Decision Tree (DT) induction algorithms. This paper provides a comprehensive approach for investigating the effects of hyperparameter tuning for the two DT induction algorithms most often used, CART and C4.5. DT induction algorithms present high predictive performance and interpretable classification models, though many HPs need to be adjusted. Experiments were carried out with different tuning strategies to induce models and to evaluate HPs' relevance using 94 classification datasets from OpenML. The experimental results point out that different HP profiles for the tuning of each algorithm provide statistically significant improvements in most of the datasets for CART, but only in one-third for C4.5. Although different algorithms may present different tuning scenarios, the tuning techniques generally required few evaluations to find accurate solutions. Furthermore, the best technique for all the algorithms was the IRACE. Finally, we found out that tuning a specific small subset of HPs is a good alternative for achieving optimal predictive performance.

Figures (13)

• Figure 1: Example of a decision tree classification. When unlabeled data is provided to the tree, conditions are applied starting from the root node and following the appropriate branch until a leaf is reached. The class is recommended based on the leaf pointed out. Adapted from Tan:2005.
• Figure 2: Experimental methodology used to adjust dt hyperparameters. The tuning is conducted via nested cross-validation: $3$-fold cv for computing fitness values and $10$-fold cv for assessing performances. The outputs are the hyperparameter settings, the predicted performances, and the optimization paths of each technique.
• Figure 3: Hyperparameter tuning results for the J48 algorithm.
• Figure 5: Scatter plot comparing performances obtained by default hyperparameter values against the optimized ones.
• Figure 6: J48 average tree size ($log_2$).