Improving Decision Trees through the Lens of Parameterized Local Search

TL;DR

This work analyzes improving decision trees via two natural local-search operations: threshold adjustment and cut exchange. It establishes NP-completeness of these problems and systematically maps the parameterized complexity landscape, showing fixed-parameter tractability when combining a small number of features $d$ with a small feature-domain size $D$, and providing DP-based FPT algorithms under the $(d+D)$ parameter. The authors connect hardness results from optimal decision-tree learning to the local-search variants, present reductions from problems like Partial Vertex Cover and Multicolored Independent Set, and supply a proof-of-concept implementation with empirical evaluation demonstrating that local search yields limited improvements on typical heuristics but can be valuable when pruning is extensive. Collectively, the results clarify when local search can meaningfully aid decision-tree construction and offer practical algorithms and insights for integrating local search with pruning in real-world settings.

Abstract

Algorithms for learning decision trees often include heuristic local-search operations such as (1) adjusting the threshold of a cut or (2) also exchanging the feature of that cut. We study minimizing the number of classification errors by performing a fixed number of a single type of these operations. Although we discover that the corresponding problems are NP-complete in general, we provide a comprehensive parameterized-complexity analysis with the aim of determining those properties of the problems that explain the hardness and those that make the problems tractable. For instance, we show that the problems remain hard for a small number $d$ of features or small domain size $D$ but the combination of both yields fixed-parameter tractability. That is, the problems are solvable in $(D + 1)^{2d} \cdot |I|^{O(1)}$ time, where $|I|$ is the size of the input. We also provide a proof-of-concept implementation of this algorithm and report on empirical results.

Figures (7)

• Figure 1: A Hasse diagram of the parameters, where an edge between two parameters means that the lower one is upper bounded by some function of the upper one. The dashed edge applies only to the Threshold Adjustment problem. The color of the left box next to each parameter indicates the complexity for Threshold Adjustment and the color of the right box for Cut Exchange, which happen to be the same for all considered single parameters. The texts of the boxes refer to the corresponding theorems, and BF stands for a brute-force algorithm. The figure shows that for example, both problems are W[1]-hard and in XP when parameterized by $s$, and consequently they are at least W[1]-hard (potentially paraNP-hard) for all parameters smaller than $s$ (for example $\ell$) and have at least an XP-time algorithm (potentially FPT) for all parameters larger than $s$ (for example $n$).
• Figure 2: Summary of our results for combinations of parameters. In each box, the center cell indicates the complexity class (and references the corresponding theorem) which holds for all parameterizations bounded from below by the parameter of the bottom cell and from above by the parameter of the top cell. As a concrete example, the first box of the middle row states that Threshold Adjustment is W$[1]$-hard and in XP for, for example, $k$, $k+D$, $k+t$, and $k+D+t$. Note that Threshold Adjustment parameterized by $t+\ell+D$ is W[1]-hard but XP-tractability is open.
• Figure 3: The gadget used for misclassifying leaf examples that do not correspond to the $j$th leaf $v$ in \ref{['thm:adjustment-to-exchange']}. Note that the class of $v$ is not relevant for this gadget.
• Figure 4: A visualization of the reduction from the proof of \ref{['thm:exc-w-hard-d-t']}. Part $a)$ shows a Multicolored Independent Set instance. For the sake of the illustration, the property that all partite sets have the same size is dropped and thus $2(p-1)\kappa+2=10$. A multicolored independent set is depicted in orange. Part $b)$ shows the corresponding classification instance. Here, $E(v_i^j)$ is the set of all edges incident with vertex $v_i^j$. $Rest_i^<$, $Rest_i^>$, $Rest^*$, and $Rest$ refers to all other examples not shown in that feature (the precise set differs in each feature and is always a subset of all examples having the default threshold of that feature). All possible cuts in the classification instance are shown by "$|$". Moreover, cuts in the input decision tree are shown in violet and cuts of the solution decision tree are shown in brown. Part $c)$ shows the input tree $T'$. Part $d)$ shows one possible solution tree $T$.
• Figure 5: A visualization of the reduction from the proof of \ref{['thm:adj-w-hard-d-t']}. Part $a)$ shows a Multicolored Independent Set instance. For the sake of the illustration, the property that all partite sets have the same size is dropped. A multicolored independent set is depicted in orange. The vertex selection in the partite set $V_1$, and the final cut in feature $d^*$. Part $b)$ shows the part of the input tree $T'$ corresponding to the vertex selection in the partite set $V_1$ together with the final cut in feature $d^*$. Part $c)$ shows the part of a solution tree $T$ corresponding to the vertex selection in the partite set $V_1$ together with the final cut in feature $d^*$. The cuts where a threshold adjustment was applied are marked orange and they indicate the selection of vertex $v_1^2\in V_1$. Part $d)$ shows the corresponding classification instance. Here, $E(v_i^j)$ is the set of all edges incident with vertex $v_i^j$. $Rest$ refers to all other examples not shown in that feature (the precise set differs in each feature and is always a subset of all examples having the default threshold of that feature). Features $x_j^<$ and $x_j^>$ for $x\in\{p,q\}$ and $j\in\{2,3\}$ are not shown. Note that thresholds $-4$ to $-1$ only exist for features $d_i^<$ and that thresholds $d_i^>$ only exist for features $d_i^>$. All possible cuts in the classification instance are shown by "$|$". Moreover, we use a 4 coloring to distinguish cuts of the initial tree $T'$ and a solution tree $T$: 1) cyan cuts are used by both $T'$ and $T$, 2) brown cuts are only used by $T'$, orange cuts are only used by $T$, and 4) black cuts are neither used by $T'$ nor $T$.

Theorems & Definitions (40)

• Theorem 4.1: $\bigstar$
• proof
• Proposition 4.3: $\bigstar$
• proof
• Proposition 4.4: $\bigstar$
• Proposition 4.5: $\bigstar$
• proof
• proof : Proof of \ref{['prop-hardness-thr-adjust']}
• Theorem 4.6: $\bigstar$
• proof