On the Depth of Monotone ReLU Neural Networks and ICNNs

TL;DR

This work analyzes depth as a fundamental resource for two restricted ReLU network families, monotone networks ($\mathrm{ReLU}^+$) and ICNNs, by linking expressivity to polyhedral geometry. Using Newton polytopes, indecomposability, and isoperimetric arguments, it derives sharp depth lower bounds and inapproximability results for exact CPWL representations. It shows MAX$_n$ cannot be computed or approximated by monotone networks, while MAX$_n$ requires depth $n$ in ICNNs, and demonstrates strong depth-separation phenomena between depth-2 ReLU networks and ICNNs. The results extend to 3D polytope constructions, establishing that shallow ICNNs cannot realize certain convex CPWL polytopes, thereby highlighting depth as a fundamental computational resource for restricted neural architectures.

Abstract

We study two models of ReLU neural networks: monotone networks (ReLU$^+$) and input convex neural networks (ICNN). Our focus is on expressivity, mostly in terms of depth, and we prove the following lower bounds. For the maximum function MAX$_n$ computing the maximum of $n$ real numbers, we show that ReLU$^+$ networks cannot compute MAX$_n$, or even approximate it. We prove a sharp $n$ lower bound on the ICNN depth complexity of MAX$_n$. We also prove depth separations between ReLU networks and ICNNs; for every $k$, there is a depth-2 ReLU network of size $O(k^2)$ that cannot be simulated by a depth-$k$ ICNN. The proofs are based on deep connections between neural networks and polyhedral geometry, and also use isoperimetric properties of triangulations.

Figures (17)

• Figure 1: . A depth $2$$\mathop{\mathrm{\mathsf{ReLU}}}\nolimits$ neural network computing the maximum of $4$ elements.
• Figure 2: . The graph of the function $F(x,y) = \mathsf{max} \{x,y,x+y,0\}$ and its Newton polytope $N(F)$ obtained as the convex hull of the four points $(0,0), (0,1), (1,0), (1,1)$. The sub-gradient $\partial F(0)$ is equal to the entire set $N(F)$.
• Figure 3: . The sub-gradient at $x$ of the function $F=\mathsf{max}\{0,x_1,x_2,x_1+x_2\}$.
• Figure 4: . The square pyramid $P_*$.
• Figure 5: . The structure of $Q'$.

Theorems & Definitions (43)

• Claim 1
• Theorem 2
• proof : Proof sketch
• Lemma 3
• Theorem 4
• Proposition 5
• Proposition 6
• Theorem 7
• Corollary 8
• Theorem 9