Understanding Deep Neural Networks with Rectified Linear Units

TL;DR

This work analyzes the expressive capacity and training complexity of ReLU DNNs. It establishes a rigorous link between ReLU networks and continuous piecewise-linear functions, proving a near-complete representation equivalence and bounding required depth for universal PWL representation. It introduces depth-based gap results showing deep nets can be exponentially more efficient than shallow ones, first via smooth 1D hard-function constructions and then through zonotope-based continua in higher dimensions. An exact training algorithm for 2-layer ReLU networks demonstrates global optimality with runtime that is polynomial in data but exponential in input dimension, highlighting fundamental complexity trade-offs. Together, these results advance understanding of when depth helps and how to approach exact learning in a nonconvex neural-network setting, with implications for theoretical foundations and algorithm design.

Abstract

In this paper we investigate the family of functions representable by deep neural networks (DNN) with rectified linear units (ReLU). We give an algorithm to train a ReLU DNN with one hidden layer to *global optimality* with runtime polynomial in the data size albeit exponential in the input dimension. Further, we improve on the known lower bounds on size (from exponential to super exponential) for approximating a ReLU deep net function by a shallower ReLU net. Our gap theorems hold for smoothly parametrized families of "hard" functions, contrary to countable, discrete families known in the literature. An example consequence of our gap theorems is the following: for every natural number $k$ there exists a function representable by a ReLU DNN with $k^2$ hidden layers and total size $k^3$, such that any ReLU DNN with at most $k$ hidden layers will require at least $\frac{1}{2}k^{k+1}-1$ total nodes. Finally, for the family of $\mathbb{R}^n\to \mathbb{R}$ DNNs with ReLU activations, we show a new lowerbound on the number of affine pieces, which is larger than previous constructions in certain regimes of the network architecture and most distinctively our lowerbound is demonstrated by an explicit construction of a *smoothly parameterized* family of functions attaining this scaling. Our construction utilizes the theory of zonotopes from polyhedral theory.

Figures (4)

• Figure 1: We fix the ${\hbox{\boldmath$\bf a$}}$ vectors for a two hidden layer $\mathbb R \rightarrow \mathbb R$ hard function as ${\hbox{\boldmath$\bf a$}} ^1 = {\hbox{\boldmath$\bf a$}} ^2 = (\frac{1}{2}) \in \Delta ^1_1$ Left: A specific hard function induced by $\ell_1$ norm: $\operatorname{ZONOTOPE}_{2,2,2}^{2}[{\hbox{\boldmath$\bf a$}} ^1, {\hbox{\boldmath$\bf a$}} ^2 , {\hbox{\boldmath$\bf b$}} ^1, {\hbox{\boldmath$\bf b$}} ^2]$ where ${\hbox{\boldmath$\bf b$}} ^1 = (0,1)$ and ${\hbox{\boldmath$\bf b$}} ^2 = (1,0)$. Note that in this case the function can be seen as a composition of $H_{{\hbox{\boldmath$\bf a$}} ^1, {\hbox{\boldmath$\bf a$}} ^2}$ with $\ell_1$-norm $N_{\ell_1}(x):=\|x\|_1 = \gamma_{Z\left((0,1),(1,0)\right)}$. Middle: A typical hard function $\operatorname{ZONOTOPE}_{2,2,4}^{2}[{\hbox{\boldmath$\bf a$}} ^1 , {\hbox{\boldmath$\bf a$}} ^2,{\hbox{\boldmath$\bf c$}} ^1,{\hbox{\boldmath$\bf c$}} ^2,{\hbox{\boldmath$\bf c$}} ^3,{\hbox{\boldmath$\bf c$}} ^4]$ with generators ${\hbox{\boldmath$\bf c$}} ^1 = (\frac{1}{4},\frac{1}{2}), {\hbox{\boldmath$\bf c$}} ^2=(-\frac{1}{2},0), {\hbox{\boldmath$\bf c$}} ^3=(0,-\frac{1}{4})$ and ${\hbox{\boldmath$\bf c$}} ^4=(-\frac{1}{4},-\frac{1}{4})$. Note how increasing the number of zonotope generators makes the function more complex. Right: A harder function from $\operatorname{ZONOTOPE}_{3,2,4}^{2}$ family with the same set of generators ${\hbox{\boldmath$\bf c$}}_1,{\hbox{\boldmath$\bf c$}}_2,{\hbox{\boldmath$\bf c$}}_3,c_4$ but one more hidden layer $(k=3)$. Note how increasing the depth make the function more complex. (For illustrative purposes we plot only the part of the function which lies above zero.)
• Figure 2: Top: $h_{{\hbox{\boldmath$\bf a$}}^1}$ with ${\hbox{\boldmath$\bf a$}}^1\in \Delta_1^2$ with 3 pieces in the range $[0,1]$. Middle: $h_{{\hbox{\boldmath$\bf a$}}^2}$ with ${\hbox{\boldmath$\bf a$}}^2 \in \Delta_1^1$ with 2 pieces in the range $[0,1]$. Bottom: $H_{{\hbox{\boldmath$\bf a$}}^1,{\hbox{\boldmath$\bf a$}}^2}=h_{{\hbox{\boldmath$\bf a$}}^2} \circ h_{{\hbox{\boldmath$\bf a$}}^1}$ with $2\cdot 3 = 6$ pieces in the range $[0,1]$. The dotted line in the bottom panel corresponds to the function in the top panel. It shows that for every piece of the dotted graph, there is a full copy of the graph in the middle panel.
• Figure 3: A 2-layer ReLU DNN computing $\max \{ x_1,x_2 \} = \frac{x_1+x_2}{2}+\frac{|x_1-x_2|}{2}$
• Figure 4: The number of pieces increasing after activation. If the blue function is $f$, then the red function $g = \max\{0,f+b\}$ has at most twice the number of pieces as $f$ for any bias $b\in \mathbb R$.

Theorems & Definitions (45)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Theorem 2.1
• Theorem 2.2
• Theorem 2.3
• Theorem 3.1
• Theorem 3.2
• Corollary 3.3