Active Learning for Decision Trees with Provable Guarantees

TL;DR

This work addresses label-efficient active learning for binary decision trees by analyzing the disagreement coefficient, showing polylogarithmic label complexity under two structural assumptions: unique feature dimensions along root-to-leaf paths and grid-like data. It introduces a general multiplicative-error active learning algorithm and applies it to decision trees, achieving a $(1+\varepsilon)$-approximate classifier with near-optimal $\varepsilon$-dependence and polylog label complexity. The authors establish an explicit upper bound $\theta = O(\ln^{d}(n))$ for decision trees of height $d$ (and provide VC-dimension bounds), along with necessity results and a weighted-uniform relaxation to broaden applicability. A lower bound for stump learning shows the multiplicative framework cannot be bypassed, and the general algorithm extends to arbitrary classifiers with label complexity depending on the disagreement coefficient and VC dimension. Together, these results bridge practical decision-tree learning with solid theoretical guarantees and suggest future work on relaxing assumptions and extending to broader data domains.

Abstract

This paper advances the theoretical understanding of active learning label complexity for decision trees as binary classifiers. We make two main contributions. First, we provide the first analysis of the disagreement coefficient for decision trees-a key parameter governing active learning label complexity. Our analysis holds under two natural assumptions required for achieving polylogarithmic label complexity, (i) each root-to-leaf path queries distinct feature dimensions, and (ii) the input data has a regular, grid-like structure. We show these assumptions are essential, as relaxing them leads to polynomial label complexity. Second, we present the first general active learning algorithm for binary classification that achieves a multiplicative error guarantee, producing a $(1+ε)$-approximate classifier. By combining these results, we design an active learning algorithm for decision trees that uses only a polylogarithmic number of label queries in the dataset size, under the stated assumptions. Finally, we establish a label complexity lower bound, showing our algorithm's dependence on the error tolerance $ε$ is close to optimal.

Figures (4)

• Figure 1: (a) A decision tree with $4$ leaves ($L=4$). Leaf $1$ uses dimensions $1, 2$ so $d_{h,1}=\{1,2\}$. (b) $\text{LineTree}_{h,3}$ classifies all samples as $1-L_{h,3}$ except those reaching leaf $3$ of $h$.
• Figure 2: (a) A decision tree that assigns label $1$, if and only if $x_a=c$. (b) A decision tree assigning label $1$ only to $X_c$ when $X_i = \langle i, i, \cdots, i \rangle$.
• Figure 3: Comparison of success rate grids for various $(c_1, b_1, c_2, b_2)$ parameterizations when running Algorithm \ref{['alg:stump_epsilin_m']} with $\delta=0.1$ (expected success rate $> 90\%$). For each cell we run Algorithm \ref{['alg:stump_epsilin_m']} on a fixed randomly generated dataset for 50 times and calculate the success rate. evidence suggests setting each of these constants to $3$ is typically sufficient for reliable algorithm performance.
• Figure : Stump algorithm

Theorems & Definitions (89)

• Theorem 1.1
• Theorem 1.2
• Corollary 1.3
• Definition 3.1: Distance & Hypothesis Ball
• Definition 3.2: Disagreement Region
• Definition 3.3: Disagreement Coefficient
• Definition 3.4: $\text{LineTree}$
• Proposition 3.5
• Theorem 3.6
• Theorem 3.7