Learning and Transferring Physical Models through Derivatives

TL;DR

This work presents Derivative Learning (DERL), a supervised approach that models physical systems by learning and matching their partial derivatives, enforcing initial and boundary conditions, and allowing empirical derivatives when analytical ones are unavailable. The authors provide theoretical guarantees showing that derivative-based learning is sufficient to recover the true solution and demonstrate superior generalization to unseen domain points and parametric PDEs, compared to learning the solution directly. A distillation-based transfer protocol leverages DERL to transfer physical knowledge across models, enabling incremental learning across time horizons, domain extensions, and PDE parameters, with higher-order derivative distillation further improving performance. The framework bridges ideas from PINNs and Sobolev learning, linking to continual learning and offering data-efficient, physically consistent models for complex dynamical systems with practical implications for modular, multi-stage model design and transfer in physics-informed ML.

Abstract

We propose Derivative Learning (DERL), a supervised approach that models physical systems by learning their partial derivatives. We also leverage DERL to build physical models incrementally, by designing a distillation protocol that effectively transfers knowledge from a pre-trained model to a student one. We provide theoretical guarantees that DERL can learn the true physical system, being consistent with the underlying physical laws, even when using empirical derivatives. DERL outperforms state-of-the-art methods in generalizing an ODE to unseen initial conditions and a parametric PDE to unseen parameters. We also design a method based on DERL to transfer physical knowledge across models by extending them to new portions of the physical domain and a new range of PDE parameters. We believe this is the first attempt at building physical models incrementally in multiple stages.

Figures (24)

• Figure 1: How to learn and transfer knowledge from a physical model. A physical model is first trained on a given domain. The model is not able to access the entire domain at once (the shaded area is unavailable). Later, the model is trained on a new portion of the domain, while previous knowledge is transferred and preserved through distillation. The same process can be applied to a time-space domain $[0,T]\times \Omega$. This way, DERL learns a solution incrementally in time. DERL implements both the learning and the transfer phase.
• Figure 2: Graphical representation on how DERL learns a function $u(t,x,y)$. Partial derivatives $\pdv{\hat{u}}{t}, \pdv{\hat{u}}{x}, \pdv{\hat{u}}{y}$ of the Neural Network (NN) output $\hat{u}$ are computed by Automatic Differentiation (AD). DERL learns the partial derivatives as independent targets, together with the initial or boundary condition $u|_{\partial\Omega}$. We compare DERL to OUTL, which learns the function $u$ directly in the domain $\Omega$ and on the boundary $\partial \Omega$.
• Figure 3: $L^2$ error difference in the learned solution $(\hat{{\bm{u}}}, \hat{p})$ between each methodology and DERL. The blue area is where DERL performs better than the comparison.
• Figure 4: Pendulum experiment. $L^2$ error difference in the learned field at $t=0$ between each methodology and DERL. The blue area is where DERL performs better than the comparison.
• Figure 5: Prediction errors for the KdV transfer experiment.

Theorems & Definitions (12)

• Theorem 3.1
• Theorem 3.2
• Theorem 3.3
• proof
• Theorem B.1: Evans2022-bl Section 5.6, theorem 3 and Section 5.8.1, theorem 1
• Theorem B.2: Mazya2011-SobolevSpaces Section 6.11.1, corollary 2
• Corollary B.3
• proof
• proof
• Remark B.4