Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RUNNs: Ritz-Uzawa Neural Networks for Solving Variational Problems

Pablo Herrera, Jamie M. Taylor, Carlos Uriarte, Ignacio Muga, David Pardo, Kristoffer G. van der Zee

Abstract

Solving Partial Differential Equations (PDEs) using neural networks presents different challenges, including integration errors and spectral bias, often leading to poor approximations. In addition, standard neural network-based methods, such as Physics-Informed Neural Networks (PINNs), often lack stability when dealing with PDEs characterized by low-regularity solutions. To address these limitations, we introduce the Ritz--Uzawa Neural Networks (RUNNs) framework, an iterative methodology to solve strong, weak, and ultra-weak variational formulations. Rewriting the PDE as a sequence of Ritz-type minimization problems within a Uzawa loop provides an iterative framework that, in specific cases, reduces both bias and variance during training. We demonstrate that the strong formulation offers a passive variance reduction mechanism, whereas variance remains persistent in weak and ultra-weak regimes. Furthermore, we address the spectral bias of standard architectures through a data-driven frequency tuning strategy. By initializing a Sinusoidal Fourier Feature Mapping based on the Normalized Cumulative Power Spectral Density (NCPSD) of previous residuals or their proxies, the network dynamically adapts its bandwidth to capture high-frequency components and severe singularities. Numerical experiments demonstrate the robustness of RUNNs, accurately resolving highly oscillatory solutions and successfully recovering a discontinuous $L^2$ solution from a distributional $H^{-2}$ source -- a scenario where standard energy-based methods fail.

RUNNs: Ritz-Uzawa Neural Networks for Solving Variational Problems

Abstract

Solving Partial Differential Equations (PDEs) using neural networks presents different challenges, including integration errors and spectral bias, often leading to poor approximations. In addition, standard neural network-based methods, such as Physics-Informed Neural Networks (PINNs), often lack stability when dealing with PDEs characterized by low-regularity solutions. To address these limitations, we introduce the Ritz--Uzawa Neural Networks (RUNNs) framework, an iterative methodology to solve strong, weak, and ultra-weak variational formulations. Rewriting the PDE as a sequence of Ritz-type minimization problems within a Uzawa loop provides an iterative framework that, in specific cases, reduces both bias and variance during training. We demonstrate that the strong formulation offers a passive variance reduction mechanism, whereas variance remains persistent in weak and ultra-weak regimes. Furthermore, we address the spectral bias of standard architectures through a data-driven frequency tuning strategy. By initializing a Sinusoidal Fourier Feature Mapping based on the Normalized Cumulative Power Spectral Density (NCPSD) of previous residuals or their proxies, the network dynamically adapts its bandwidth to capture high-frequency components and severe singularities. Numerical experiments demonstrate the robustness of RUNNs, accurately resolving highly oscillatory solutions and successfully recovering a discontinuous solution from a distributional source -- a scenario where standard energy-based methods fail.
Paper Structure (51 sections, 4 theorems, 61 equations, 12 figures, 4 tables, 1 algorithm)

This paper contains 51 sections, 4 theorems, 61 equations, 12 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Mathematical Framework
  3. Model problem
  4. The Uzawa method
  5. Convergence of the Uzawa method
  6. Combining Ritz and Uzawa: three approaches
  7. Approach 1
  8. Approach 2
  9. Approach 3
  10. Simplifications under Different Problem Formulations
  11. Strong Formulation
  12. Weak formulation
  13. Ultra-weak formulation
  14. Practical Setup: Inexact Uzawa Method
  15. Inexact Approach 1.
  16. ...and 36 more sections

Key Result

Proposition 1

Let $\left\{u^k\right\}\subset\mathbb U$ be a sequence and define $\left\{r^k\right\}:=\left\{r(u^k)\right\}\subset \mathbb V$ and $\left\{\delta^k\right\}=\left\{\delta(r^k)\right\}\subset\mathbb U$. Then, the following statements are equivalent:

Figures (12)

  • Figure 1: Experimental results with Adam: (a) Relative error and (b) The NCPSD analysis.
  • Figure 2: Analysis of the correction step ($k=0$) after $6,000$ epochs. Top: Comparison between the true error $e^0$ and the learned correction $r^0$. Bottom: Comparison of their derivatives.
  • Figure 3: Final error function.
  • Figure 4: Experimental results with LS/Adam: (a) Relative error and (b) The NCPSD analysis.
  • Figure 5: Analysis of the correction step ($k=0$) after $3,000$ epochs. Top: Comparison between the true error $e^0$ and the learned correction $r^0$. Bottom: Comparison of their derivatives.
  • ...and 7 more figures

Theorems & Definitions (9)

  • Proposition 1
  • proof
  • Proposition 2
  • proof
  • Remark 1: Ideal convergence with graph inner product
  • Theorem 1
  • proof
  • Theorem 2
  • proof