Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReLU Deep Neural Networks and Linear Finite Elements

Juncai He, Lin Li, Jinchao Xu, Chunyue Zheng

TL;DR

This work establishes a precise link between ReLU deep neural networks and continuous piecewise linear functions, with a focus on those arising from linear finite elements. It proves that any CPWL function in ℝ^d can be represented by a ReLU DNN with depth at most ⌈log2(d+1)⌉, and shows that linear finite element functions require at least 2 hidden layers when d≥2, with 2 layers optimal for d=2,3. It also provides constructive representations for LFE basis functions, analyzes error behavior against adaptive FEM, and explores low-bit-width DNN models as a viable representation for CPWL functions. The paper then extends the discussion to numerical PDEs, introducing a DNN-Galerkin method and presenting a 1D example where the DNN-based approach surpasses adaptive FEM with the same degrees of freedom. Overall, the results offer a theoretical foundation for the expressive power of deep ReLU networks, justify certain quantized architectures, and suggest practical PDE-solving strategies that blend FEM insights with DNN methods.

Abstract

In this paper, we investigate the relationship between deep neural networks (DNN) with rectified linear unit (ReLU) function as the activation function and continuous piecewise linear (CPWL) functions, especially CPWL functions from the simplicial linear finite element method (FEM). We first consider the special case of FEM. By exploring the DNN representation of its nodal basis functions, we present a ReLU DNN representation of CPWL in FEM. We theoretically establish that at least $2$ hidden layers are needed in a ReLU DNN to represent any linear finite element functions in $Ω\subseteq \mathbb{R}^d$ when $d\ge2$. Consequently, for $d=2,3$ which are often encountered in scientific and engineering computing, the minimal number of two hidden layers are necessary and sufficient for any CPWL function to be represented by a ReLU DNN. Then we include a detailed account on how a general CPWL in $\mathbb R^d$ can be represented by a ReLU DNN with at most $\lceil\log_2(d+1)\rceil$ hidden layers and we also give an estimation of the number of neurons in DNN that are needed in such a representation. Furthermore, using the relationship between DNN and FEM, we theoretically argue that a special class of DNN models with low bit-width are still expected to have an adequate representation power in applications. Finally, as a proof of concept, we present some numerical results for using ReLU DNNs to solve a two point boundary problem to demonstrate the potential of applying DNN for numerical solution of partial differential equations.

ReLU Deep Neural Networks and Linear Finite Elements

TL;DR

This work establishes a precise link between ReLU deep neural networks and continuous piecewise linear functions, with a focus on those arising from linear finite elements. It proves that any CPWL function in ℝ^d can be represented by a ReLU DNN with depth at most ⌈log2(d+1)⌉, and shows that linear finite element functions require at least 2 hidden layers when d≥2, with 2 layers optimal for d=2,3. It also provides constructive representations for LFE basis functions, analyzes error behavior against adaptive FEM, and explores low-bit-width DNN models as a viable representation for CPWL functions. The paper then extends the discussion to numerical PDEs, introducing a DNN-Galerkin method and presenting a 1D example where the DNN-based approach surpasses adaptive FEM with the same degrees of freedom. Overall, the results offer a theoretical foundation for the expressive power of deep ReLU networks, justify certain quantized architectures, and suggest practical PDE-solving strategies that blend FEM insights with DNN methods.

Abstract

In this paper, we investigate the relationship between deep neural networks (DNN) with rectified linear unit (ReLU) function as the activation function and continuous piecewise linear (CPWL) functions, especially CPWL functions from the simplicial linear finite element method (FEM). We first consider the special case of FEM. By exploring the DNN representation of its nodal basis functions, we present a ReLU DNN representation of CPWL in FEM. We theoretically establish that at least hidden layers are needed in a ReLU DNN to represent any linear finite element functions in when . Consequently, for which are often encountered in scientific and engineering computing, the minimal number of two hidden layers are necessary and sufficient for any CPWL function to be represented by a ReLU DNN. Then we include a detailed account on how a general CPWL in can be represented by a ReLU DNN with at most hidden layers and we also give an estimation of the number of neurons in DNN that are needed in such a representation. Furthermore, using the relationship between DNN and FEM, we theoretically argue that a special class of DNN models with low bit-width are still expected to have an adequate representation power in applications. Finally, as a proof of concept, we present some numerical results for using ReLU DNNs to solve a two point boundary problem to demonstrate the potential of applying DNN for numerical solution of partial differential equations.

Paper Structure

This paper contains 24 sections, 18 theorems, 137 equations, 7 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Deep neural network (DNN) generated by ReLU
  3. General DNN
  4. A shallow neural network ${\rm DNN}_1$
  5. Linear finite element (LFE) function as a DNN
  6. Linear finite element spaces
  7. DNN representation of finite element functions
  8. Comparison of error estimates in adaptive finite element and DNN methods
  9. LFE can not be recovered by ${\rm DNN}_1$ for $d\ge 2$
  10. General CPWL as a ReLU DNN
  11. The main result
  12. On the proof of Theorem \ref{['main0']}
  13. Linear finite element functions as DNN with $\lceil\log_2 (d+1)\rceil$ hidden layers
  14. Low bit-width DNN models
  15. Application to Numerical PDEs
  16. ...and 9 more sections

Key Result

Lemma 2.1

Every ReLU DNN: $\mathbb{R}^d\to\mathbb{R}^c$ is a continuous piecewise linear function. More specifically, given any ReLU DNN, there is a polyhedral decomposition of $\mathbb R^d$ such that this ReLU DNN is linear on each polyhedron in such a decomposition.

Figures (7)

  • Figure 2.1: Projections of the domain partitions formed by 2-layer ReLU DNNs with sizes $(n_0, n_1, n_2, n_3)= (2, 5, 5, 1), (2, 10, 10, 1) \text{and} (2, 20, 20, 1)$ with random parameters.
  • Figure 3.1: an interval, a triangle and a tedrohedron partition
  • Figure 3.2: The basis function in 1D and 2D
  • Figure 3.3: Left: $x\in G(i)$, right: $x\notin G(i)$
  • Figure 7.1: The grid of AFEM and DNN(the right figure is the top of left).
  • ...and 2 more figures

Theorems & Definitions (20)

  • Lemma 2.1
  • Theorem 2.1
  • Lemma 3.1
  • Remark 1
  • Theorem 3.1
  • Corollary 3.1
  • Theorem 4.1
  • Corollary 4.1
  • Lemma 5.1
  • Theorem 5.1
  • ...and 10 more