Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Memorization capacity of deep ReLU neural networks characterized by width and depth

Xin Yang, Yunfei Yang

TL;DR

This paper investigates the minimal size of such networks to memorize any $N$ data points in the unit ball with pairwise separation distance $\delta$ and discrete labels and proves that any such networks should also satisfy the lower bound, which implies that the construction is optimal up to logarithmic factors when $\delta^{-1}$ is polynomial in $N$.

Abstract

This paper studies the memorization capacity of deep neural networks with ReLU activation. Specifically, we investigate the minimal size of such networks to memorize any $N$ data points in the unit ball with pairwise separation distance $δ$ and discrete labels. Most prior studies characterize the memorization capacity by the number of parameters or neurons. We generalize these results by constructing neural networks, whose width $W$ and depth $L$ satisfy $W^2L^2= \mathcal{O}(N\log(δ^{-1}))$, that can memorize any $N$ data samples. We also prove that any such networks should also satisfy the lower bound $W^2L^2=Ω(N \log(δ^{-1}))$, which implies that our construction is optimal up to logarithmic factors when $δ^{-1}$ is polynomial in $N$. Hence, we explicitly characterize the trade-off between width and depth for the memorization capacity of deep neural networks in this regime.

Memorization capacity of deep ReLU neural networks characterized by width and depth

TL;DR

This paper investigates the minimal size of such networks to memorize any data points in the unit ball with pairwise separation distance and discrete labels and proves that any such networks should also satisfy the lower bound, which implies that the construction is optimal up to logarithmic factors when is polynomial in .

Abstract

This paper studies the memorization capacity of deep neural networks with ReLU activation. Specifically, we investigate the minimal size of such networks to memorize any data points in the unit ball with pairwise separation distance and discrete labels. Most prior studies characterize the memorization capacity by the number of parameters or neurons. We generalize these results by constructing neural networks, whose width and depth satisfy , that can memorize any data samples. We also prove that any such networks should also satisfy the lower bound , which implies that our construction is optimal up to logarithmic factors when is polynomial in . Hence, we explicitly characterize the trade-off between width and depth for the memorization capacity of deep neural networks in this regime.
Paper Structure (6 sections, 11 theorems, 54 equations)

This paper contains 6 sections, 11 theorems, 54 equations.

Table of Contents

  1. Introduction
  2. Main result
  3. Constructive Proof
  4. Proof of Lemma \ref{['Lemma 2.6']}
  5. Optimality
  6. Conclusion

Key Result

Theorem 2.1

Let $N, d, C \in \mathbb{N}$, $C\ge 2$, $\delta > 0$ and denote $R := 10 N^2 \delta^{-1} \sqrt{\pi d}$. Consider a collection of $N$ labeled samples $({\boldsymbol{x}}_1, y_1), \dots, ({\boldsymbol{x}}_N, y_N) \in \mathbb{R}^d \times \mathbb{R}$ satisfying the following two conditions: Then, for any $S,T \in \mathbb{N}$ with $S<N$, there exists a neural network $F : \mathbb{R}^d \to \mathbb{R}$ w

Theorems & Definitions (16)

  • Theorem 2.1
  • Proposition 2.2
  • Lemma 2.3
  • Lemma 2.4
  • Lemma 2.5
  • proof
  • Lemma 2.6
  • proof
  • Lemma 2.7
  • Lemma 2.8
  • ...and 6 more