Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Preconditioning and Numerical Stability in Neural Network Training for Parametric PDEs

Markus Bachmayr, Wolfgang Dahmen, Chenguang Duan, Mathias Oster

TL;DR

The paper addresses training neural surrogates for parameter-dependent PDEs and the numerical instability that arises from standard preconditioned representations. It introduces stable frame-based representations and a matrix factorization $\mathbf H^T \mathbf A_y \mathbf H = \mathbf D^T \mathbf C_y \mathbf D$ to enable accurate, low-precision computations ($\sim$ single- and half-precision). The authors show that combining variational formulations with multilevel frame layouts (e.g., BPX) yields well-conditioned mappings and improved optimization convergence, particularly for Adam, while providing matrix-free implementations to boost efficiency. This work offers practical guidance for neural operator learning on low-precision hardware without sacrificing discretization-level accuracy.

Abstract

In the context of training neural network-based approximations of solutions of parameter-dependent PDEs, we investigate the effect of preconditioning via well-conditioned frame representations of operators and demonstrate a significant improvement on the performance of standard training methods. We also observe that standard representations of preconditioned matrices are insufficient for obtaining numerical stability and propose a generally applicable form of stable representations that enables computations with single- and half-precision floating point numbers without loss of precision.

Preconditioning and Numerical Stability in Neural Network Training for Parametric PDEs

TL;DR

The paper addresses training neural surrogates for parameter-dependent PDEs and the numerical instability that arises from standard preconditioned representations. It introduces stable frame-based representations and a matrix factorization to enable accurate, low-precision computations ( single- and half-precision). The authors show that combining variational formulations with multilevel frame layouts (e.g., BPX) yields well-conditioned mappings and improved optimization convergence, particularly for Adam, while providing matrix-free implementations to boost efficiency. This work offers practical guidance for neural operator learning on low-precision hardware without sacrificing discretization-level accuracy.

Abstract

In the context of training neural network-based approximations of solutions of parameter-dependent PDEs, we investigate the effect of preconditioning via well-conditioned frame representations of operators and demonstrate a significant improvement on the performance of standard training methods. We also observe that standard representations of preconditioned matrices are insufficient for obtaining numerical stability and propose a generally applicable form of stable representations that enables computations with single- and half-precision floating point numbers without loss of precision.
Paper Structure (17 sections, 3 theorems, 59 equations, 5 figures, 3 tables)

This paper contains 17 sections, 3 theorems, 59 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Problem formulation
  3. Choice of basis, preconditioning, and performance of optimization schemes
  4. Stable representations and relation to prior works
  5. Norm stability of representation systems and condition numbers
  6. Examplary variational formulations
  7. Riesz basis and frame representations
  8. Multilevel frames
  9. Effect of preconditioning on convergence of optimization schemes
  10. Effect on random initializations
  11. Numerical stability of preconditioned problems
  12. Stable representations
  13. Network structures
  14. Matrix-free numerical realization
  15. Experimental settings
  16. ...and 2 more sections

Key Result

Theorem 3

Let $\lVert \varphi_{j,k} \rVert_{H^1(\Omega)} \eqsim 1$. Then $\{ \varphi_{j,k} \colon j = 1, \ldots, J , \; k \in \mathcal{I}_j \}$ is a frame for $W_J \subset H^1(\Omega)$ with frame constants independent of $J$.

Figures (5)

  • Figure 1: Random initialization with and without frame representation. (Top) Initialization with frame representation. (Bottom) Initialization without frame representation.
  • Figure 2: Different neural network architectures mapping from the parameter space to the frame coefficients. Without loss of generality, we take three-levels of frame as an example. (Top) Full low-rank ResNet. (Middle) Separate low-rank ResNet. (Bottom) Separate frame representation.
  • Figure 3: Loss curves of FOSLS with and without preconditioning using different optimizers under different computing precisions.
  • Figure 4: Loss curves of numerically stable preconditioning using different optimizers under different computing precisions.
  • Figure 5: Loss curves of numerically stable preconditioning with different network architectures under different computing precisions.

Theorems & Definitions (8)

  • Example 1
  • Example 2
  • Theorem 3
  • Proposition 4
  • proof
  • Corollary 5
  • Example 6
  • Example 7