Approximating splits for decision trees quickly in sparse data streams
Nikolaj Tatti
TL;DR
The paper tackles fast split selection in streaming decision trees with sparse binary features and binary labels. It introduces two near-optimal, approximation-based strategies, UpdEnt for information gain and UpdGini for the Gini index, that bucket features by conditional probabilities and maintain per-bin structures to achieve a $(1 + \alpha)$-approximation with amortized times of $O(\alpha^{-1}(1+m\log d)\log\log n)$ for entropy and $O(\alpha^{-1} + m\log d)$ for Gini. The methods leverage sparsity ($m \ll d$) to deliver sublinear update-time performance, and experiments on synthetic and benchmark datasets show substantial speedups over exact baselines while preserving near-optimal splits. Overall, the work enables efficient, real-time decision-tree learning in sparse data streams.
Abstract
Decision trees are one of the most popular classifiers in the machine learning literature. While the most common decision tree learning algorithms treat data as a batch, numerous algorithms have been proposed to construct decision trees from a data stream. A standard training strategy involves augmenting the current tree by changing a leaf node into a split. Here we typically maintain counters in each leaf which allow us to determine the optimal split, and whether the split should be done. In this paper we focus on how to speed up the search for the optimal split when dealing with sparse binary features and a binary class. We focus on finding splits that have the approximately optimal information gain or Gini index. In both cases finding the optimal split can be done in $O(d)$ time, where $d$ is the number of features. We propose an algorithm that yields $(1 + α)$ approximation when using conditional entropy in amortized $O(α^{-1}(1 + m\log d) \log \log n)$ time, where $m$ is the number of 1s in a data point, and $n$ is the number of data points. Similarly, for Gini index, we achieve $(1 + α)$ approximation in amortized $O(α^{-1} + m \log d)$ time. Our approach is beneficial for sparse data where $m \ll d$. In our experiments we find almost-optimal splits efficiently, faster than the baseline, overperforming the theoretical approximation guarantees.