Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Regularized dynamical parametric approximation of stiff evolution problems

Christian Lubich, Jörg Nick

TL;DR

This work addresses the challenge of evolving irregular nonlinear parametrizations, such as neural networks, in stiff evolution problems by developing regularized parametric time integrators. It introduces a regulated Dirac–Frenkel–type approach where the parameter time derivatives are obtained from a regularized nonlinear least-squares problem, and the time stepping uses implicit Euler, implicit midpoint, and Runge--Kutta methods solved via Gauss--Newton iterations. The authors establish error bounds linking the parametric iterates to the nonparametric Newton scheme, derive global error estimates that are robust to stiffness, and extend the framework to higher-order RK methods like Radau IIA and Gauß. Numerical experiments on transport and heat equations illustrate defect decay and long-time stability, while implementation details highlight practical considerations for quadrature, initialization, and adaptive regularization. Overall, the paper provides a principled, stiffness-robust methodology for simulating high-dimensional, irregular parametrizations in evolution equations with provable error control and practical algorithmic guidance.

Abstract

Evolutionary deep neural networks have emerged as a rapidly growing field of research. This paper studies numerical integrators for such and other classes of nonlinear parametrizations $ u(t) = Φ(θ(t)) $, where the evolving parameters $θ(t)$ are to be computed. The primary focus is on tackling the challenges posed by the combination of stiff evolution problems and irregular parametrizations, which typically arise with neural networks, tensor networks, flocks of evolving Gaussians, and in further cases of overparametrization. We propose and analyse regularized parametric versions of the implicit Euler method and higher-order implicit Runge--Kutta methods for the time integration of the parameters in nonlinear approximations to evolutionary partial differential equations and large systems of stiff ordinary differential equations. At each time step, an ill-conditioned nonlinear optimization problem is solved approximately with a few regularized Gauss--Newton iterations. Error bounds for the resulting parametric integrator are derived by relating the computationally accessible Gauss--Newton iteration for the parameters to the computationally inaccessible Newton iteration for the underlying non-parametric time integration scheme. The theoretical findings are supported by numerical experiments that are designed to show key properties of the proposed parametric integrators.

Regularized dynamical parametric approximation of stiff evolution problems

TL;DR

This work addresses the challenge of evolving irregular nonlinear parametrizations, such as neural networks, in stiff evolution problems by developing regularized parametric time integrators. It introduces a regulated Dirac–Frenkel–type approach where the parameter time derivatives are obtained from a regularized nonlinear least-squares problem, and the time stepping uses implicit Euler, implicit midpoint, and Runge--Kutta methods solved via Gauss--Newton iterations. The authors establish error bounds linking the parametric iterates to the nonparametric Newton scheme, derive global error estimates that are robust to stiffness, and extend the framework to higher-order RK methods like Radau IIA and Gauß. Numerical experiments on transport and heat equations illustrate defect decay and long-time stability, while implementation details highlight practical considerations for quadrature, initialization, and adaptive regularization. Overall, the paper provides a principled, stiffness-robust methodology for simulating high-dimensional, irregular parametrizations in evolution equations with provable error control and practical algorithmic guidance.

Abstract

Evolutionary deep neural networks have emerged as a rapidly growing field of research. This paper studies numerical integrators for such and other classes of nonlinear parametrizations , where the evolving parameters are to be computed. The primary focus is on tackling the challenges posed by the combination of stiff evolution problems and irregular parametrizations, which typically arise with neural networks, tensor networks, flocks of evolving Gaussians, and in further cases of overparametrization. We propose and analyse regularized parametric versions of the implicit Euler method and higher-order implicit Runge--Kutta methods for the time integration of the parameters in nonlinear approximations to evolutionary partial differential equations and large systems of stiff ordinary differential equations. At each time step, an ill-conditioned nonlinear optimization problem is solved approximately with a few regularized Gauss--Newton iterations. Error bounds for the resulting parametric integrator are derived by relating the computationally accessible Gauss--Newton iteration for the parameters to the computationally inaccessible Newton iteration for the underlying non-parametric time integration scheme. The theoretical findings are supported by numerical experiments that are designed to show key properties of the proposed parametric integrators.
Paper Structure (30 sections, 9 theorems, 104 equations, 14 figures)

This paper contains 30 sections, 9 theorems, 104 equations, 14 figures.

Table of Contents

  1. Introduction
  2. Mathematical setting
  3. Evolution problem
  4. Evolving parametrization
  5. Parametric implicit Euler and midpoint methods
  6. Reference point: the implicit Euler method
  7. Regularized parametric implicit Euler method
  8. Variants
  9. Implicit midpoint rule
  10. Regularized Gauß--Newton iteration
  11. Error bound
  12. Preparation: Bounds for the non-parametric implicit Euler method
  13. Local error
  14. Global error
  15. Parametric implicit Runge--Kutta methods
  16. ...and 15 more sections

Key Result

theorem 1

In the setting of Section sec:setting and under a restriction between the stepsize $h$ and the regularization parameter $\varepsilon$, the regularized Gauß--Newton iteration gn-1 with $J_0=A$ yields iterates $u_1^{k+1}=\Phi(\theta_1^{k+1})$ that deviate from the Newton iterates $y_1^{k+1}\in{\@fontswitch\mathcal{H}}$ of newton by where $C=1+\beta c$ with the bound $\beta$ of the second derivativ

Figures (14)

  • Figure 1: Lady Windermere's fan for the global error $y_n-y(t_n)$ of the non-parametric (semi-discrete) implicit Euler method \ref{['implicit-euler']}. Each arrow symbolizes a step of the implicit Euler method. The vertical lines illustrate local errors, which are then propagated by the implicit Euler method. The global error is the sum of the propagated local errors.
  • Figure 2: Lady Windermere's fan for the estimation of the global difference $\|u_n-y_n\|$, which measures the effect of the regularized parametrization on the implicit time integration method. Arrows denote a single time step of the underlying (semi-discrete) time integration scheme.
  • Figure 3: Different initial functions, which are used for the numerical experiments of the transport equation and the heat equation in one dimension.
  • Figure 4: Errors of the initial neural network approximations shown in Figure \ref{['fig:initfuns']}. Most of the error is localized around points of low regularity.
  • Figure 5: Sorted weights of the corresponding neural network approximations. The absolute value of the parameters is bounded by a very moderate bound, despite the shallow network architecture and the high curvature at several points of the approximation.
  • ...and 9 more figures

Theorems & Definitions (23)

  • theorem 1: Iteration error
  • proof
  • remark thmcounterremark: Stepsize vs. regularization restriction
  • remark thmcounterremark: Variants of regularization
  • remark thmcounterremark: Adaptive regularization parameter
  • remark thmcounterremark: Parametric implicit midpoint rule
  • lemma thmcounterlemma: Local error
  • proof
  • lemma thmcounterlemma: Stable error propagation by the implicit Euler method
  • theorem 2: Global error bound
  • ...and 13 more