Superfast Selection for Decision Tree Algorithms

TL;DR

This work addresses the bottleneck of split selection in decision trees on large tabular data by introducing Superfast Selection, which reduces the per-feature split evaluation from $O(MN)$ to $O(M)$ using a one-pass scan, intermediate statistics, and a prefix-sum approach. It extends this method to Ultrafast Decision Tree (UDT) by integrating into CART, pre-sorting numerical values at initialization, and enabling Training Only Once Tuning to avoid repeated training during hyper-parameter search. The key contributions include a general framework that handles numerical, categorical, and hybrid features without pre-encoding, rigorous complexity analyses ($O(M)$ per feature, $\Theta(K M \log M)$ average training, $O(K M^2)$ worst-case), and substantial empirical speedups across large datasets with improved memory efficiency. The practical impact includes enabling real-time, edge-device learning and more efficient hyper-parameter tuning for tabular data, with potential implications for IoT, mobile deployments, and latency-sensitive applications.

Abstract

We present a novel and systematic method, called Superfast Selection, for selecting the "optimal split" for decision tree and feature selection algorithms over tabular data. The method speeds up split selection on a single feature by lowering the time complexity, from O(MN) (using the standard selection methods) to O(M), where M represents the number of input examples and N the number of unique values. Additionally, the need for pre-encoding, such as one-hot or integer encoding, for feature value heterogeneity is eliminated. To demonstrate the efficiency of Superfast Selection, we empower the CART algorithm by integrating Superfast Selection into it, creating what we call Ultrafast Decision Tree (UDT). This enhancement enables UDT to complete the training process with a time complexity O(KM$^2$) (K is the number of features). Additionally, the Training Only Once Tuning enables UDT to avoid the repetitive training process required to find the optimal hyper-parameter. Experiments show that the UDT can finish a single training on KDD99-10% dataset (494K examples with 41 features) within 1 second and tuning with 214.8 sets of hyper-parameters within 0.25 second on a laptop.