Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Introduction to optimization methods for training SciML models

Alena Kopaničáková, Elisa Riccietti

TL;DR

The paper provides a unified overview of optimization methods for both ML and SciML, highlighting that SciML losses driven by PDEs and operators are globally coupled and often ill-conditioned, contrasting with the data-driven locality of classical ML losses. It connects training dynamics to the neural tangent kernel, explains spectral bias and stiffness, and surveys first-order, second-order, and hybrid methods, including SGD theory, sampling strategies for PINNs, and preconditioning approaches. Practical guidance is offered through numerical examples and tutorial-style discussions of regu-larization, loss balancing, adaptive sampling, and switching between optimization regimes. The work underscores open research directions at the interface of scientific computing and machine learning, aiming to develop scalable, curvature-aware, and physics-consistent optimization techniques for increasingly complex SciML applications.

Abstract

Optimization is central to both modern machine learning (ML) and scientific machine learning (SciML), yet the structure of the underlying optimization problems differs substantially across these domains. Classical ML typically relies on stochastic, sample-separable objectives that favor first-order and adaptive gradient methods. In contrast, SciML often involves physics-informed or operator-constrained formulations in which differential operators induce global coupling, stiffness, and strong anisotropy in the loss landscape. As a result, optimization behavior in SciML is governed by the spectral properties of the underlying physical models rather than by data statistics, frequently limiting the effectiveness of standard stochastic methods and motivating deterministic or curvature-aware approaches. This document provides a unified introduction to optimization methods in ML and SciML, emphasizing how problem structure shapes algorithmic choices. We review first- and second-order optimization techniques in both deterministic and stochastic settings, discuss their adaptation to physics-constrained and data-driven SciML models, and illustrate practical strategies through tutorial examples, while highlighting open research directions at the interface of scientific computing and scientific machine learning.

Introduction to optimization methods for training SciML models

TL;DR

The paper provides a unified overview of optimization methods for both ML and SciML, highlighting that SciML losses driven by PDEs and operators are globally coupled and often ill-conditioned, contrasting with the data-driven locality of classical ML losses. It connects training dynamics to the neural tangent kernel, explains spectral bias and stiffness, and surveys first-order, second-order, and hybrid methods, including SGD theory, sampling strategies for PINNs, and preconditioning approaches. Practical guidance is offered through numerical examples and tutorial-style discussions of regu-larization, loss balancing, adaptive sampling, and switching between optimization regimes. The work underscores open research directions at the interface of scientific computing and machine learning, aiming to develop scalable, curvature-aware, and physics-consistent optimization techniques for increasingly complex SciML applications.

Abstract

Optimization is central to both modern machine learning (ML) and scientific machine learning (SciML), yet the structure of the underlying optimization problems differs substantially across these domains. Classical ML typically relies on stochastic, sample-separable objectives that favor first-order and adaptive gradient methods. In contrast, SciML often involves physics-informed or operator-constrained formulations in which differential operators induce global coupling, stiffness, and strong anisotropy in the loss landscape. As a result, optimization behavior in SciML is governed by the spectral properties of the underlying physical models rather than by data statistics, frequently limiting the effectiveness of standard stochastic methods and motivating deterministic or curvature-aware approaches. This document provides a unified introduction to optimization methods in ML and SciML, emphasizing how problem structure shapes algorithmic choices. We review first- and second-order optimization techniques in both deterministic and stochastic settings, discuss their adaptation to physics-constrained and data-driven SciML models, and illustrate practical strategies through tutorial examples, while highlighting open research directions at the interface of scientific computing and scientific machine learning.
Paper Structure (59 sections, 4 theorems, 100 equations, 11 figures, 4 algorithms)

This paper contains 59 sections, 4 theorems, 100 equations, 11 figures, 4 algorithms.

Table of Contents

  1. Introduction
  2. Empirical risk minimization and the finite sum minimization problem
  3. Examples of ERM across learning paradigms
  4. Example 1: Supervised learning - classification
  5. Logistic regression (linear model)
  6. Deep neural network (nonlinear model)
  7. Example 2: Supervised learning - regression
  8. Example 3: Physics-informed neural networks (PINNs)
  9. Enhancing the problem definition
  10. Regularization
  11. Balancing the loss terms
  12. Bilevel approaches
  13. Adaptive weighting strategies
  14. Solving the optimization problem
  15. Neural tangent kernel
  16. ...and 44 more sections

Key Result

Theorem 1

Let assumptions hp_smooth, hp_convex and hp_noise hold, and assume that $f$ is $\mu$-strongly convex. Assume that $\{\theta_k\}_{k\in\mathbb{N}}$ is the sequence generated by SGD eq:sgd with a constant step size $0<\alpha_k=\alpha< \frac{1}{2 L_{\max}}$ for all $k$. Then, for $k\geq 1$, one has

Figures (11)

  • Figure 1: Left: Example of binary classification in the $(x_1,x_2)$-plane: Samples from two classes $y_i \in \{0,1\}$ are shown as orange diamonds and blue circles, respectively. The dotted line indicates the decision boundary separating the two regions. Right: Example of a computational domain $\Omega$ with sampled collocation points: Dirichlet boundaries $\Gamma_D$ (blue), Neumann boundaries $\Gamma_N$ (orange), interior points $\pazocal{D}_\Omega$ (purple), and optional empirical data samples $\pazocal{D}_{\text{data}}$ (yellow).
  • Figure 2: Example of spectral bias to learn target function $u_{\theta}$ (purple) composed of low-, medium-, and high-frequencies. The network prediction $u_{\theta}$ (purple) is reported at training iterations 100 (left), 1500 (middle), and 7500 (right).
  • Figure 3: Comparison of the convergence behaviors of GD, the adaptive gradient method (Adam), and the Gauss-Newton (GN) method. Left: Evolution of the function value over the iterations. Right: Condition numbers of the corresponding empirical kernels $\kappa(\Theta_k^{(M_k)})$.
  • Figure 4: Two-dimensional projection li2018visualizing of the objective function landscape for the same DNN under two training regimes. Left: Data-driven nonlinear regression. Right: PINN.
  • Figure 5: Trajectories of GD (blue) and SGD on a smooth quadratic function for different noise levels $\sigma \in \{\sigma_0,\;\sigma_0/2,\;\sigma_0/4\}$ (purple, orange, yellow). Higher noise magnitudes enlarge the region in which SGD oscillates around the minimizer $\theta^\ast$, while lower noise confines the iterates closer to the GD path.
  • ...and 6 more figures

Theorems & Definitions (8)

  • Definition 1
  • Definition 2
  • Definition 3
  • Theorem 1
  • proof
  • Theorem 2
  • Theorem 3
  • Theorem 4