Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Curse of Dimensionality in Neural Network Optimization

Sanghoon Na, Haizhao Yang

Abstract

This paper demonstrates that when a shallow neural network with a Lipschitz continuous activation function is trained using either empirical or population risk to approximate a target function that is $r$ times continuously differentiable on $[0,1]^d$, the population risk may not decay at a rate faster than $t^{-\frac{4r}{d-2r}}$, where $t$ denotes the time parameter of the gradient flow dynamics. This result highlights the presence of the curse of dimensionality in the optimization computation required to achieve a desired accuracy. Instead of analyzing parameter evolution directly, the training dynamics are examined through the evolution of the parameter distribution under the 2-Wasserstein gradient flow. Furthermore, it is established that the curse of dimensionality persists when a locally Lipschitz continuous activation function is employed, where the Lipschitz constant in $[-x,x]$ is bounded by $O(x^δ)$ for any $x \in \mathbb{R}$. In this scenario, the population risk is shown to decay at a rate no faster than $t^{-\frac{(4+2δ)r}{d-2r}}$. Understanding how function smoothness influences the curse of dimensionality in neural network optimization theory is an important and underexplored direction that this work aims to address.

Curse of Dimensionality in Neural Network Optimization

Abstract

This paper demonstrates that when a shallow neural network with a Lipschitz continuous activation function is trained using either empirical or population risk to approximate a target function that is times continuously differentiable on , the population risk may not decay at a rate faster than , where denotes the time parameter of the gradient flow dynamics. This result highlights the presence of the curse of dimensionality in the optimization computation required to achieve a desired accuracy. Instead of analyzing parameter evolution directly, the training dynamics are examined through the evolution of the parameter distribution under the 2-Wasserstein gradient flow. Furthermore, it is established that the curse of dimensionality persists when a locally Lipschitz continuous activation function is employed, where the Lipschitz constant in is bounded by for any . In this scenario, the population risk is shown to decay at a rate no faster than . Understanding how function smoothness influences the curse of dimensionality in neural network optimization theory is an important and underexplored direction that this work aims to address.

Paper Structure

This paper contains 22 sections, 19 theorems, 98 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Related Works
  3. Mean-field theory of neural networks and Wasserstein gradient flows
  4. Barron spaces
  5. Setup
  6. Main Theorems
  7. Key Lemmas
  8. Growth of second moments under the 2-Wasserstein gradient flow
  9. Barron norms and second moments
  10. Slow approximation property across infinitely many time scales
  11. Low complexity estimates
  12. Curse of dimensionality in numerical integration
  13. Proofs of the Main Theorems
  14. Proof of Theorem \ref{['poor']}
  15. Proof of Theorem \ref{['global']}
  16. ...and 7 more sections

Key Result

Theorem 1.1

Let the training samples be independent and identically distributed from the uniform distribution on $[0,1]^d.$ Let $\sigma:\mathbb{R}\to\mathbb{R}$ be a Lipschitz continuous activation function and $r$ be a positive integer with $r<d/2.$ There exists a target function $\phi\in C^r([0,1]^d)$ such th holds for all $\gamma > \frac{4r}{d-2r}.$ Here, $f_t$ denotes the shallow neural network at trainin

Figures (1)

  • Figure 1: Geometric description of the proof of Theorem \ref{['global']}. The green curve with arrow illustrates the sublinear growth of the Barron norm, which follows from Lemma \ref{['Second moment evolution']} and Lemma \ref{['Control of Barron norm']}. The shallow neural network at the initialization is denoted as $f_{\pi^t}$, represented as a circle filled with dark blue. The shallow neural network training time $t$ is denoted as $f_{\pi^t}$, represented as a circle filled with sky-blue. The black dotted line represents Theorem \ref{['poor']}, the existence of $C^r(Q)$ function with slow approximation property.

Theorems & Definitions (36)

  • Theorem 1.1
  • Theorem 4.1
  • Corollary 4.2
  • Theorem 4.3
  • Theorem 4.4
  • Lemma 5.1: wojtowytsch2020convergence
  • Lemma 5.2
  • Lemma 5.3
  • Definition 5.1
  • Lemma 5.4
  • ...and 26 more