A Depth Hierarchy for Computing the Maximum in ReLU Networks via Extremal Graph Theory

TL;DR

This work addresses the problem of exactly computing $\operatorname{Max}_d$ with ReLU networks and proves a depth–width hierarchy: for depth $k$ with $3 \le k \le \log_2(\log_2 d)$, any network implementing $\operatorname{Max}_d$ on $[0,1]^d$ must have width at least $0.1 d^{1+\frac{1}{2^{k-2}-1}}$. The authors develop a novel combinatorial proof that links the first-layer non-linearities to a graph on input coordinates and then invoke Turán's theorem to force a large clique, enabling a dimension-reduction and layer-collapse induction across depths. The lower bound is unconditional and scales super-linearly with $d$, contrasting with prior approximation results and highlighting the intrinsic geometric complexity of the non-differentiable ridges of $\operatorname{Max}_d$. The approach provides a new avenue for proving depth-based lower bounds for exact CPWL computations in ReLU networks, and clarifies the fundamental power of depth beyond constant-depth regimes.

Abstract

We consider the problem of exact computation of the maximum function over $d$ real inputs using ReLU neural networks. We prove a depth hierarchy, wherein width $Ω\big(d^{1+\frac{1}{2^{k-2}-1}}\big)$ is necessary to represent the maximum for any depth $3\le k\le \log_2(\log_2(d))$. This is the first unconditional super-linear lower bound for this fundamental operator at depths $k\ge3$, and it holds even if the depth scales with $d$. Our proof technique is based on a combinatorial argument and associates the non-differentiable ridges of the maximum with cliques in a graph induced by the first hidden layer of the computing network, utilizing Turán's theorem from extremal graph theory to show that a sufficiently narrow network cannot capture the non-linearities of the maximum. This suggests that despite its simple nature, the maximum function possesses an inherent complexity that stems from the geometric structure of its non-differentiable hyperplanes, and provides a novel approach for proving lower bounds for deep neural networks.

Figures (2)

• Figure 3: The construction of the graph $G_{\mathcal{N}}$ that is induced by the first-hidden-layer weights of $\mathcal{N}$. Left: The complete graph $K_5$ where edges are colored according to the first-layer neuron $\mathbf{w}_i$ that removes them. Middle: The weight vectors of the first hidden layer whose non-zero coordinates dictate the edge removal process. Right: The final graph $G_{\mathcal{N}}$ after the removal of all colored edges. If the width $n$ is small relative to $d$, Turán's theorem ensures that the graph still contains a large clique (e.g., the triangle formed by the first three coordinates).
• Figure 4: An illustration of the effects of the negative assignment of values explained in Step 4 (Subsection \ref{['subsec:step4']}). By constructing a clique from the first three coordinates, we ensure that every neuron in the first hidden layer possesses a non-zero weight for some coordinate $j \ge 4$. Since the clique coordinates are bounded in $[0,1]$, assigning sufficiently large negative values to $x_4$ and $x_5$ ensures that each neuron's pre-activation is dominated by the term corresponding to its largest non-zero weight index. Consequently, these operating ranges (highlighted in red) remain strictly within either the positive or negative rays of the ReLU, rendering the first hidden layer's non-linearities redundant. Crucially, as Step 2 (Subsection \ref{['subsec:step2']}) guarantees that the network computes $\text{Max}_d$ globally, this negative assignment forces the maximum to be attained by one of the first three coordinates. This effectively reduces the network's computation to $\text{Max}_3$ on the clique's domain, facilitating the lower bound proof.

Theorems & Definitions (13)

• Theorem 1.1
• Theorem 1.2: Turán's Theorem turan1941extremal
• Theorem 2.1
• Lemma A.1
• proof
• Proposition A.2
• proof
• Lemma A.3
• proof
• Proposition A.4