Universal Differential Equations for Scientific Machine Learning

TL;DR

The paper introduces universal differential equations (UDEs) and the SciML ecosystem as a unified approach to blend physical laws with data-driven models, enabling efficient training, stable adjoint-based gradients, and scalable solutions for high-dimensional and stochastic systems. It demonstrates knowledge-guided symbolic regression, high-dimensional PDE solving via USDEs, and automated closure learning across geophysical and rheological contexts, achieving substantial speedups and data efficiency. Key contributions include a comprehensive adjoint framework, extensive benchmarking against standard ML-DE tools, and practical demonstrations of equation discovery, model reduction, and closure parameterization. Collectively, these advances offer a scalable, physics-informed toolkit for scientific machine learning with broad applicability across engineering, climate science, and materials modeling.

Abstract

In the context of science, the well-known adage "a picture is worth a thousand words" might well be "a model is worth a thousand datasets." In this manuscript we introduce the SciML software ecosystem as a tool for mixing the information of physical laws and scientific models with data-driven machine learning approaches. We describe a mathematical object, which we denote universal differential equations (UDEs), as the unifying framework connecting the ecosystem. We show how a wide variety of applications, from automatically discovering biological mechanisms to solving high-dimensional Hamilton-Jacobi-Bellman equations, can be phrased and efficiently handled through the UDE formalism and its tooling. We demonstrate the generality of the software tooling to handle stochasticity, delays, and implicit constraints. This funnels the wide variety of SciML applications into a core set of training mechanisms which are highly optimized, stabilized for stiff equations, and compatible with distributed parallelism and GPU accelerators.

Figures (16)

• Figure 1: Simplified flow chart of the SciML software ecosystem. The orange boxes on the left denote high level use cases, from PDE-constrained optimal control and acausal modeling to direct construction of UDEs and other forms of differential equations for numerical solving. The white boxes denote the ModelingToolkit.jl symbolic-numeric system ma2021modelingtoolkit used for automated construction of symbolically-optimized numerical code. These symbolic tools are built on the Symbolics.jl computer algebra system gowda2021high developed and maintained by the SciML developers. The symbolic tools or direct user input generate numerical descriptions of the mathematical problems to solve, shown in the orange triangles. The OptimizationProblem structs are consumed by the GalacticOptim.jl global and local optimization library while the differential equation models of any form are consumed by DifferentialEquations.jl DifferentialEquations.jl-2017. These models can be composed and stacked, i.e. OptimizationProblems containing ODEProblems. These tools then use underlying numerical solvers, blue denoting part of the SciML ecosystem, purple libraries denoting external libraries developed and maintained by SciML developers, red denoting external libraries used and contributed to by SciML developers.
• Figure 2: Automated Lotka-Volterra equation discovery with UODE-enhanced SINDy. The known and unknown parts of the model are trained against the data (A), using neural networks to reduce the complexity of the unknown equations. Afterwards symbolic regression is used to generate a fully symbolic model. (B) shows the estimation of the true polynomial interactions (black) and the neural network (red), (C) the resulting error over time.
• Figure 3: The extrapolation of the knowledge-enhanced SINDy fit series. The green and yellow dots show the data that was used to fit the UODE, and the dots show the true solution of the Lotka-Volterra Equations \ref{['eq:LV']} beyond the training data. The blue and purple lines show the extrapolated solution how the UODE-enhanced SINDy recovered equations.
• Figure 4: Recovery of the UPDE for the Fisher-KPP equation. (A) Training data and (B) prediction of the UPDE for $\rho(x,t)$. (C) Curves for the weights of the CNN filter $[w_1, w_2, w_3]$ indicate the recovery of the $[1, -2, 1]$ stencil for the 1-dimensional Laplacian. (D) Comparison of the learned (blue) and the true growth term (orange) showcases the learned parabolic form of the missing nonlinear equation.
• Figure 5: Convergence of neural closure relations for a non-Newtonian Fluid. (A) Error between the approximated $\sigma$ using the linear approximation Equation \ref{['eq:OldroydB']} and the neural network closure relation Equation \ref{['eq:nnsigma']} against the full FENE-P solution. The error is measured for the strain rates $\dot{\gamma} = 12\cos \omega t$ for $\omega = 1,1.2,\ldots,2$ and tested with the strain rate $\dot{\gamma} = 12\cos 1.5 t$. (B) Predictions of stress for testing strain rate for the linear approximation and UODE solution against the exact FENE-P stress.