A Gap Between Decision Trees and Neural Networks

TL;DR

The paper investigates when geometric simplicity of axis-aligned decision boundaries (interpreted as interpretability) conflicts with accurate approximation by shallow ReLU networks. It uses the Radon total variation $\|\cdot\|_{\mathcal{R}}$ to quantify boundary complexity and shows that hard threshold indicators $1_A$ have infinite $\mathcal{R}$-TV, with many common smoothings retaining this pathology in dimension $d>1$, while Gaussian smoothing yields a finite, dimension-dependent bound. A key positive result is that for classification tasks focusing only on thresholding, one can construct a smooth barrier score $S_B$ that exactly recovers the box at a fixed threshold and admits calibration guarantees with finite $\mathcal{R}$TV. However, when learning a calibrated score that remains geometrically simple, there is a sharp trade-off between accuracy and complexity, revealing an intrinsic gap between decision-tree interpretability and score-level learning in higher dimensions. The experiments on synthetic box unions illustrate how threshold tuning can alter the decision set without changing the learned score, highlighting the practical implication that threshold-focused evaluations may mask underlying score geometry.

Abstract

We study when geometric simplicity of decision boundaries, used here as a notion of interpretability, can conflict with accurate approximation of axis-aligned decision trees by shallow neural networks. Decision trees induce rule-based, axis-aligned decision regions (finite unions of boxes), whereas shallow ReLU networks are typically trained as score models whose predictions are obtained by thresholding. We analyze the infinite-width, bounded-norm, single-hidden-layer ReLU class through the Radon total variation ($\mathrm{R}\mathrm{TV}$) seminorm, which controls the geometric complexity of level sets. We first show that the hard tree indicator $1_A$ has infinite $\mathrm{R}\mathrm{TV}$. Moreover, two natural split-wise continuous surrogates--piecewise-linear ramp smoothing and sigmoidal (logistic) smoothing--also have infinite $\mathrm{R}\mathrm{TV}$ in dimensions $d>1$, while Gaussian convolution yields finite $\mathrm{R}\mathrm{TV}$ but with an explicit exponential dependence on $d$. We then separate two goals that are often conflated: classification after thresholding (recovering the decision set) versus score learning (learning a calibrated score close to $1_A$). For classification, we construct a smooth barrier score $S_A$ with finite $\mathrm{R}\mathrm{TV}$ whose fixed threshold $τ=1$ exactly recovers the box. Under a mild tube-mass condition near $\partial A$, we prove an $L_1(P)$ calibration bound that decays polynomially in a sharpness parameter, along with an explicit $\mathrm{R}\mathrm{TV}$ upper bound in terms of face measures. Experiments on synthetic unions of rectangles illustrate the resulting accuracy--complexity tradeoff and how threshold selection shifts where training lands along it.

Figures (10)

• Figure 1: Width–$\mathcal{R}\mathrm{TV}$ frontier (depth-1 ReLU, box task).
• Figure 2: Box-classification dataset and decision-tree baseline.(a) Labeled samples for the synthetic box task, visualized on the $(x_0,x_1)$ slice with $x_2=x_3=x_4=0.5$. Points are uniformly sampled from $[0,1]^5$ and colored by box membership; the dashed rectangle denotes the ground-truth boundary on the slice. (b) An axis-aligned decision tree fit on the same data, visualized via its induced slice boundary. (c) shows how the $\mathcal{R}\mathrm{TV}$ of shallow ReLU networks grows with width on the same box task, highlighting the trade-off between approximation quality and Radon total variation. For depth-1 ReLU networks trained with Adam (with weight decay) on the $d=10$ box classification task, where $X \sim \mathrm{Unif}([0,1]^{10})$ and $y = \mathbf{1}\{x \text{ lies in a centered axis-aligned box}\}$ (roughly $50\%$ positives). Curves correspond to test raw-MSE targets $0.20, 0.25, 0.30,$ and $0.40$; for each width $m\in\{8,16,32,64,128,256,512\}$ we plot the mean $\mathcal{R}\mathrm{TV}(W,a)=\tfrac{1}{2}(\|W\|_F^2+\|a\|_2^2)$ at the first epoch when the test raw MSE drops below the target, averaged over multiple random initializations.
• Figure 3: Plot visualizes how ramp, sigmoidal, and Gaussian smoothings soften a one-dimensional hard threshold.
• Figure 4: Smooth score that thresholds exactly to an axis-aligned box. We visualize the construction $S_B(x)=\prod_{j=1}^d \vartheta_{\lambda,\varepsilon}(u_j-x_j)\,\vartheta_{\lambda,\varepsilon}(x_j-\ell_j)\in[0,1]$ for a box $B=\prod_{j=1}^d[\ell_j,u_j]\subset[0,1]^d$, where $\vartheta_{\lambda,\varepsilon}(t)=(1-h_\varepsilon(t))\,e^{\lambda\min(t,0)}+h_\varepsilon(t), \qquad h_\varepsilon(t)=H\!((t+\varepsilon)/\varepsilon),$ with $H\in C^\infty(\mathbb{R})$ a monotone cutoff satisfying $H(s)=0$ for $s\le 0$ and $H(s)=1$ for $s\ge 1$ (with vanishing endpoint derivatives), and $\varepsilon=c_0/\lambda$. (a) 1D profile: $S_B(x)=1$ on $B$ and decreases smoothly outside. (b) 2D heatmap of $S_B$ showing a plateau at $1$ on $B$ and smooth variation outside. (c) 2D slice of the same score in $d=5$ (holding the remaining coordinates fixed inside $B$). In all panels, the exact classification cutoff is $\tau=1$, i.e., $\{x:\ S_B(x)\ge 1\}=B;$ the solid white rectangle is the true box boundary on the displayed slice, and the dashed contour marks the numerical level set $S_B(x)=1-10^{-12}$.
• Figure 5: Raw logits on a 2-D slice across widths (MSE training). Heatmaps show the learned score $f_\theta(x)$ (raw logit output) on the $(x_0,x_1)$ slice with $x_2=x_3=x_4=0.5$. Row M shows the trained model output. Row O overlays the IoU-optimal decision boundary $\{x:f_\theta(x)=\tau^\star\}$, where $\tau^\star$ is selected by grid search on a held-out validation split. The dashed rectangle is the ground-truth box boundary on the slice. Panel headers report $\mathrm{IoU}@\tau{=}0 \rightarrow \mathrm{IoU}^\star$ and $\tau^\star$; note that $\mathrm{IoU}@0.5$ corresponds here to the level set at logit$0$ (since no sigmoid/probability is used). Across widths, $\mathcal{R}\mathrm{TV}$ increases: $24.1$ ($W{=}8$), $39.6$ ($16$), $119.7$ ($32$), $189.2$ ($64$), $264.2$ ($128$), $374.8$ ($256$), illustrating a fit--complexity trade-off.

Theorems & Definitions (20)

• Theorem 1: Theorem 3.1 savarese19a
• Lemma 2
• Theorem 3
• Remark 4: Implication for shallow networks
• Proposition 5: Proposition 5-(a) of Ongie2020A
• Theorem 6
• Theorem 7
• Lemma 8: Single box: exact thresholding and $L_1(P)$ control
• Theorem 9: $\mathcal{R}\mathrm{TV}$ upper bound for a single box
• Lemma 10