Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Solving High-Dimensional PDEs Using Linearized Neural Networks

Tong Mao, Jinchao Xu, Xiaofeng Xu

TL;DR

This work analyzes linearized shallow neural networks with fixed hidden-layer parameters for solving high-dimensional PDEs, comparing variational (Galerkin) and collocation (least-squares) formulations. It reveals that variational systems suffer from severe ill-conditioning as network width grows, while collocation remains numerically stable and often more accurate when paired with robust solvers. The study demonstrates that deterministic neuron designs—via quasi-uniform grids and quasi-Monte Carlo points for ReLU$^k$, and Petrushev/sphere-based schemes for $\tanh$—can match or exceed the accuracy of random features, even in high dimensions. Overall, conditioning is identified as the main bottleneck for Galerkin methods, while collocation offers a practical, scalable alternative with strong empirical performance, motivating future preconditioning and analysis work.

Abstract

Linearized shallow neural networks that are constructed by fixing the hidden-layer parameters have recently shown strong performance in solving partial differential equations (PDEs). Such models, widely used in the random feature method (RFM) and extreme learning machines (ELM), transform network training into a linear least-squares problem. In this paper, we conduct a numerical study of the variational (Galerkin) and collocation formulations for these linearized networks. Our numerical results reveal that, in the variational formulation, the associated linear systems are severely ill-conditioned, forming the primary computational bottleneck in scaling the neural network size, even when direct solvers are employed. In contrast, collocation methods combined with robust least-squares solvers exhibit better numerical stability and achieve higher accuracy as we increase neuron numbers. This behavior is consistently observed for both ReLU$^k$ and $\tanh$ activations, with $\tanh$ networks exhibiting even worse conditioning. Furthermore, we demonstrate that random sampling of the hidden layer parameters, commonly used in RFM and ELM, is not necessary for achieving high accuracy. For ReLU$^k$ activations, this follows from existing theory and is verified numerically in this paper, while for $\tanh$ activations, we introduce two deterministic schemes that achieve comparable accuracy.

Solving High-Dimensional PDEs Using Linearized Neural Networks

TL;DR

This work analyzes linearized shallow neural networks with fixed hidden-layer parameters for solving high-dimensional PDEs, comparing variational (Galerkin) and collocation (least-squares) formulations. It reveals that variational systems suffer from severe ill-conditioning as network width grows, while collocation remains numerically stable and often more accurate when paired with robust solvers. The study demonstrates that deterministic neuron designs—via quasi-uniform grids and quasi-Monte Carlo points for ReLU, and Petrushev/sphere-based schemes for —can match or exceed the accuracy of random features, even in high dimensions. Overall, conditioning is identified as the main bottleneck for Galerkin methods, while collocation offers a practical, scalable alternative with strong empirical performance, motivating future preconditioning and analysis work.

Abstract

Linearized shallow neural networks that are constructed by fixing the hidden-layer parameters have recently shown strong performance in solving partial differential equations (PDEs). Such models, widely used in the random feature method (RFM) and extreme learning machines (ELM), transform network training into a linear least-squares problem. In this paper, we conduct a numerical study of the variational (Galerkin) and collocation formulations for these linearized networks. Our numerical results reveal that, in the variational formulation, the associated linear systems are severely ill-conditioned, forming the primary computational bottleneck in scaling the neural network size, even when direct solvers are employed. In contrast, collocation methods combined with robust least-squares solvers exhibit better numerical stability and achieve higher accuracy as we increase neuron numbers. This behavior is consistently observed for both ReLU and activations, with networks exhibiting even worse conditioning. Furthermore, we demonstrate that random sampling of the hidden layer parameters, commonly used in RFM and ELM, is not necessary for achieving high accuracy. For ReLU activations, this follows from existing theory and is verified numerically in this paper, while for activations, we introduce two deterministic schemes that achieve comparable accuracy.
Paper Structure (24 sections, 3 theorems, 50 equations, 21 figures, 9 tables)

This paper contains 24 sections, 3 theorems, 50 equations, 21 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Theoretical results
  3. ReLU$^k$ shallow neural network
  4. Tanh shallow neural networks
  5. Methodology
  6. Variational Method
  7. Collocation method
  8. Numerical experiments
  9. ReLU$^k$ Shallow Neural Networks
  10. Continuous $L^2$-minimization
  11. Discrete $\ell^2$-regression
  12. Condition numbers and numerical instability
  13. Second-order elliptic PDEs
  14. Deterministic features using quasi-Monte Carlo points in high dimensions
  15. Tanh networks and deterministic schemes
  16. ...and 9 more sections

Key Result

Theorem 1

SX:2024 Given $u \in \mathcal{K}_1(\mathbb{D})$, there exists a positive constant $M$ depending on $k$ and $d$ such that where $\Sigma_{n, M}(\mathbb{D})$ and $\mathcal{K}_1(\mathbb{D})$ are the shallow neural network function space, and the variation space with respect to the $\text{ReLU}^k$ dictionary $\mathbb{D}$, respectively, and $C$ is a constant independent of $u$ and $n$.

Figures (21)

  • Figure 1: Error decay for continuous $L^2$-minimization (variational formulation) in dimensions $d=1$--$6$.
  • Figure 2: Error decay for discrete $\ell^2$-regression (collocation formulation) in dimensions $d=1$--$6$.
  • Figure 3: Condition numbers of the mass matrices for ReLU$^1$ in dimensions $d=1$--$6$. The dashed lines show least-squares fits on a log--log scale.
  • Figure 4: Condition numbers of the mass matrices for ReLU$^2$ in dimensions $d=1$--$6$. The dashed lines show least-squares fits on a log--log scale.
  • Figure 5: Error decay plots for continuous $L^2$-minimization with ReLU$^2$ in one dimension. The error decay becomes unstable as the number of neurons increases.
  • ...and 16 more figures

Theorems & Definitions (8)

  • Theorem 1
  • Theorem 2
  • Remark 1
  • Remark 2
  • Remark 3
  • Theorem 3
  • proof
  • Remark 4: Numerical advantages of solving the least–squares form