Trending in Numerical Analysis
Introduction to optimization methods for training SciML models
Optimization is central to both modern machine learning (ML) and scientific machine learning (SciML), yet the structure of the underlying optimization problems differs substantially across these domains. Classical ML typically relies on stochastic, sample-separable objectives that favor first-order and adaptive gradient methods. In contrast, SciML often involves physics-informed or operator-constrained formulations in which differential operators induce global coupling, stiffness, and strong anisotropy in the loss landscape. As a result, optimization behavior in SciML is governed by the spectral properties of the underlying physical models rather than by data statistics, frequently limiting the effectiveness of standard stochastic methods and motivating deterministic or curvature-aware approaches. This document provides a unified introduction to optimization methods in ML and SciML, emphasizing how problem structure shapes algorithmic choices. We review first- and second-order optimization techniques in both deterministic and stochastic settings, discuss their adaptation to physics-constrained and data-driven SciML models, and illustrate practical strategies through tutorial examples, while highlighting open research directions at the interface of scientific computing and scientific machine learning.
2601.10222
Jan 2026Numerical Analysis
2512.11737A Fully Discrete Surface Finite Element Method for the Navier--Stokes equations on Evolving Surfaces with prescribed Normal Velocity
We analyze two fully time-discrete numerical schemes for the incompressible Navier-Stokes equations posed on evolving surfaces in $\mathbb{R}^3$ with prescribed normal velocity using the evolving surface finite element method (ESFEM). We employ generalized Taylor-Hood finite elements $\mathrm{\mathbf{P}}_{k_u}$-- $\mathrm{P}_{k_{pr}}$-- $\mathrm{P}_{k_λ}$, $k_u=k_{pr}+1 \geq 2$, $k_λ\geq 1$, for the spatial discretization, where the normal velocity constraint is enforced weakly via a Lagrange multiplier $λ$, and a backward Euler discretization for the time-stepping procedure. Depending on the approximation order of $λ$ and weak formulation of the Navier-Stokes equations, we present stability and error analysis for two different discrete schemes, whose difference lies in the geometric information needed. We establish optimal velocity $L^{2}_{a_h}$-norm error bounds ($a_h$ an energy norm) for both schemes when $k_λ=k_u$, but only for the more information intensive one when $k_λ=k_u-1$, using iso-parametric and super-parametric discretizations, respectively, with the help of a newly derived surface Ritz-Stokes projection. Similarly, stability and optimal convergence for the pressures is established in an $L^2_{L^2}\times L^2_{H_h^{-1}}$-norm ($H_h^{-1}$ a discrete dual space) when $k_λ=k_u$, using a novel Leray time-projection to ensure weakly divergence conformity for our discrete velocity solution at two different time-steps (surfaces). Assuming further regularity conditions for the more information intensive scheme, along with an almost weak divergence conformity result at two different time-steps, we establish optimal $L^2_{L^2}\times L^2_{L^2}$-norm pressure error bounds when $k_λ=k_u-1$, using super-parametric approximation. Simulations verifying our results are provided, along with a comparison test against a penalty approach.
2512.11737
Dec 2025Numerical Analysis
Basis Choices for Frequency Domain Statistical Independence Tests and Algorithms for Algebraic Relation Extraction
In this paper, we explore how different selections of basis functions impact the efficacy of frequency domain techniques in statistical independence tests, and study different algorithms for extracting low-dimensional algebraic relations from dependent data. We examine a range of complete orthonormal bases functions including the Legendre polynomials, Fourier series, Walsh functions, and standard and nonstandard Haar wavelet bases. We utilize fast transformation algorithms to efficiently transform physical domain data to frequency domain coefficients. The main focuses of this paper are the effectiveness of different basis selections in detecting data dependency using frequency domain data, e.g., whether varying basis choices significantly influence statistical power loss for small data with large noise; and on the stability of different optimization formulations for finding proper algebraic relations when data are dependent. We present numerical results to demonstrate the effectiveness of frequency domain-based statistical analysis methods and provide guidance for selecting the proper basis and algorithm to detect a particular type of relations.
2512.01963
Dec 2025Numerical Analysis
