Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benefits of depth in neural networks

Matus Telgarsky

TL;DR

The paper demonstrates that depth fundamentally enhances the expressive power of neural networks by constructing explicit deep networks with Θ(k^3) layers, constant width, and constant parameter count that cannot be approximated by shallow networks unless the latter are exponentially large. It develops an oscillation-counting framework based on the crossing number to quantify the capacity for high-frequency behavior, showing deep networks can realize exponentially many oscillations while shallow ones cannot. The results extend beyond standard ReLU networks to the broader class of semi-algebraic gates, including convolutional nets with max and activation gates, sum-product networks, and boosted decision trees, and are complemented by VC-dimension bounds that limit the prevalence of hard labelings. Together, these findings formalize depth as a core structural advantage and connect neural-depth hierarchies to classical circuit complexity, while outlining limitations and avenues for further exploration.

Abstract

For any positive integer $k$, there exist neural networks with $Θ(k^3)$ layers, $Θ(1)$ nodes per layer, and $Θ(1)$ distinct parameters which can not be approximated by networks with $\mathcal{O}(k)$ layers unless they are exponentially large --- they must possess $Ω(2^k)$ nodes. This result is proved here for a class of nodes termed "semi-algebraic gates" which includes the common choices of ReLU, maximum, indicator, and piecewise polynomial functions, therefore establishing benefits of depth against not just standard networks with ReLU gates, but also convolutional networks with ReLU and maximization gates, sum-product networks, and boosted decision trees (in this last case with a stronger separation: $Ω(2^{k^3})$ total tree nodes are required).

Benefits of depth in neural networks

TL;DR

The paper demonstrates that depth fundamentally enhances the expressive power of neural networks by constructing explicit deep networks with Θ(k^3) layers, constant width, and constant parameter count that cannot be approximated by shallow networks unless the latter are exponentially large. It develops an oscillation-counting framework based on the crossing number to quantify the capacity for high-frequency behavior, showing deep networks can realize exponentially many oscillations while shallow ones cannot. The results extend beyond standard ReLU networks to the broader class of semi-algebraic gates, including convolutional nets with max and activation gates, sum-product networks, and boosted decision trees, and are complemented by VC-dimension bounds that limit the prevalence of hard labelings. Together, these findings formalize depth as a core structural advantage and connect neural-depth hierarchies to classical circuit complexity, while outlining limitations and avenues for further exploration.

Abstract

For any positive integer , there exist neural networks with layers, nodes per layer, and distinct parameters which can not be approximated by networks with layers unless they are exponentially large --- they must possess nodes. This result is proved here for a class of nodes termed "semi-algebraic gates" which includes the common choices of ReLU, maximum, indicator, and piecewise polynomial functions, therefore establishing benefits of depth against not just standard networks with ReLU gates, but also convolutional networks with ReLU and maximization gates, sum-product networks, and boosted decision trees (in this last case with a stronger separation: total tree nodes are required).

Paper Structure

This paper contains 17 sections, 20 theorems, 29 equations, 2 figures.

Table of Contents

  1. Setting and main results
  2. Main result
  3. Companion results
  4. Semi-algebraic gates and assorted network notation
  5. Notation for neural networks
  6. Benefits of depth
  7. Approximation via oscillation counting
  8. Few layers, few oscillations
  9. Many layers, many oscillations
  10. Proof of \ref{['fact:main']}
  11. Limitations of depth
  12. Bibliographic notes and open problems
  13. Deferred proofs
  14. Deferred proofs from \ref{['sec:sa']}
  15. Deferred proofs from \ref{['sec:apx']}
  16. ...and 2 more sections

Key Result

Theorem 1.1

Let any integer $k \geq 1$ and any dimension $d\geq 1$ be given. There exists $f:\mathbb R^d \to \mathbb R$ computed by a neural network with standard ReLU gates in $2k^3+8$ layers, $3k^3 + 12$ total nodes, and $4+d$ distinct parameters so that where $\mathcal{C}$ is the union of the following two sets of functions.

Figures (2)

  • Figure 1: $f$ crosses more than $g$.
  • Figure 2: Three partitions.

Theorems & Definitions (23)

  • Theorem 1.1
  • Theorem 1.2
  • Lemma 1.3: Simplification of \ref{['fact:pp_vc']}
  • Definition 2.1
  • Definition 2.2
  • Lemma 2.3: Example semi-algebraic gates
  • Lemma 3.1
  • Lemma 3.2
  • Lemma 3.3
  • Lemma 3.4
  • ...and 13 more