Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss

Yahong Yang, Juncai He

TL;DR

The paper addresses the depth-versus-width trade-off for neural networks under Sobolev losses, decomposing the optimal generalization error into approximation and sampling components and deriving bounds that depend on the parameter count $W$, sample size $M$, and loss-derivative order $k$. By comparing DeNNs (deep, narrow) and WeNNs (shallow, wide) with total parameter budget $W=\\mathcal{O}(N^2L)$, the authors show DeNNs excel when data is plentiful or higher-order derivatives are required, while WeNNs perform better with limited samples or many parameters; this framework extends to Sobolev losses $H^k$ with $k=0,1,2$ and to PDE solvers via deep Ritz and PINN methods. They provide near-optimal generalization bounds, validate the theory through PDE-based applications, and discuss refinement via Rademacher complexity for different regimes. The findings offer practical guidance for selecting network architectures in Sobolev training and PDE-informed learning, with future work aimed at broader function spaces and CNNs.

Abstract

Constructing the architecture of a neural network is a challenging pursuit for the machine learning community, and the dilemma of whether to go deeper or wider remains a persistent question. This paper explores a comparison between deeper neural networks (DeNNs) with a flexible number of layers and wider neural networks (WeNNs) with limited hidden layers, focusing on their optimal generalization error in Sobolev losses. Analytical investigations reveal that the architecture of a neural network can be significantly influenced by various factors, including the number of sample points, parameters within the neural networks, and the regularity of the loss function. Specifically, a higher number of parameters tends to favor WeNNs, while an increased number of sample points and greater regularity in the loss function lean towards the adoption of DeNNs. We ultimately apply this theory to address partial differential equations using deep Ritz and physics-informed neural network (PINN) methods, guiding the design of neural networks.

Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss

TL;DR

The paper addresses the depth-versus-width trade-off for neural networks under Sobolev losses, decomposing the optimal generalization error into approximation and sampling components and deriving bounds that depend on the parameter count , sample size , and loss-derivative order . By comparing DeNNs (deep, narrow) and WeNNs (shallow, wide) with total parameter budget , the authors show DeNNs excel when data is plentiful or higher-order derivatives are required, while WeNNs perform better with limited samples or many parameters; this framework extends to Sobolev losses with and to PDE solvers via deep Ritz and PINN methods. They provide near-optimal generalization bounds, validate the theory through PDE-based applications, and discuss refinement via Rademacher complexity for different regimes. The findings offer practical guidance for selecting network architectures in Sobolev training and PDE-informed learning, with future work aimed at broader function spaces and CNNs.

Abstract

Constructing the architecture of a neural network is a challenging pursuit for the machine learning community, and the dilemma of whether to go deeper or wider remains a persistent question. This paper explores a comparison between deeper neural networks (DeNNs) with a flexible number of layers and wider neural networks (WeNNs) with limited hidden layers, focusing on their optimal generalization error in Sobolev losses. Analytical investigations reveal that the architecture of a neural network can be significantly influenced by various factors, including the number of sample points, parameters within the neural networks, and the regularity of the loss function. Specifically, a higher number of parameters tends to favor WeNNs, while an increased number of sample points and greater regularity in the loss function lean towards the adoption of DeNNs. We ultimately apply this theory to address partial differential equations using deep Ritz and physics-informed neural network (PINN) methods, guiding the design of neural networks.
Paper Structure (34 sections, 36 theorems, 152 equations, 1 figure, 3 tables)

This paper contains 34 sections, 36 theorems, 152 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Organization of the paper
  3. Preliminaries
  4. Neural networks
  5. Sobolev spaces and covering number
  6. Generalization Error in $H^k$ Loss Functions for Supervised Learning
  7. Notations in Sobolev training
  8. Main results
  9. Proof Sketche of Theorem \ref{['generalzation l2']}
  10. Applications for Solving Partial Differential Equations
  11. Generalization Error in deep Ritz Method
  12. Generalization error of Physics-Informed Neural Network Methods
  13. Improving Generalization Error Across Parameter Counts
  14. Conclusions and Future Works
  15. Acknowledgments and Statements
  16. ...and 19 more sections

Key Result

Theorem 3.2

Let $d,L,M\in\mathbb{N}_+$, $B,C_1,C_2Here, the occurrences of $C_1$ and $C_2$ in $\mathcal{F}_{B,k}$ continue to hold throughout the rest of the propositions and theorems in this paper.\in\mathbb{R}_+$. For any $f\in W^{n,\infty}([0,1]^d)$ with $\|f\|_{W^{n,\infty}([0,1]^d)}\le 1$ for $n>k$ and $k= where $W=\mathcal{O}(L(\log L)^3)$ is the number of parameters in DeNNs, $\mathbf{E}$ is expected r

Figures (1)

  • Figure 1: Generalization error with respect to the number of parameters, $W$, and sample points, $M$, for $L^2$ loss functions, $H^1$ loss functions, $H^2$ loss functions, and $H^k$ loss functions with $k\ge 3$ ($W=\mathcal{O}(N^2L)$, where $N$ is the width of NNs and $L$ is the depth of NNs).

Theorems & Definitions (74)

  • Definition 2.1: Sobolev Spaces evans2022partial
  • Definition 2.2: covering number anthony1999neural
  • Definition 2.3: Uniform covering number anthony1999neural
  • Remark 3.1
  • Theorem 3.2
  • Remark 3.3
  • Remark 3.4
  • Corollary 3.5
  • proof
  • Remark 3.6
  • ...and 64 more