Physics-informed fine-tuning of foundation models for partial differential equations

Abstract

Foundation models for partial differential equations (PDEs) have emerged as powerful surrogates pre-trained on diverse physical systems, but adapting them to new downstream tasks remains challenging due to limited task-specific data and distribution shifts. While fine-tuning has proven transformative in natural language processing, best practices for adapting PDE foundation models remain underexplored. Although physics-informed training has successfully trained accurate solvers across a wide range of PDE problems, its potential for fine-tuning data-based foundation models has not been systematically studied. In this work, we introduce a physics-informed fine-tuning framework that adapts pre-trained PDE foundation models by incorporating physical constraints (PDE residuals and boundary conditions) directly into the fine-tuning objective. This enables effective adaptation in data-scarce regimes while promoting physical consistency. We evaluate our method on a downstream task composed of an unseen PDE class and compare it with data-driven finetuning counterparts. Our results demonstrate that physics-informed fine-tuning achieves competitive accuracy without requiring PDE solutions for training. Furthermore, a hybrid fine-tuning strategy yields superior generalization to out-of-distribution scenarios when only minimal training data is available. These findings establish physics-informed fine-tuning as a scalable and data-efficient paradigm, providing a physically interpretable pathway for adapting foundation models in scientific machine learning.

Figures (6)

• Figure 1: Physics-informed fine-tuning of pre-trained foundation model with scalable Operator Transformer architecture herde2024poseidon for the Poisson downstream task (cf. Section \ref{['ssc:benchmark']}).
• Figure 2: Number of labeled training samples $M$ vs. median relative $L^1$ error of the predicted solution compared to the ground-truth solution across 240 test samples from the training distribution (left) and across 50 out-of-distribution test samples (right).
• Figure 3: Interpolation performance: qualitative comparison of predicted solutions for test samples from the training distribution. Number of labeled training samples $M$: 2048. Batch size: 40.
• Figure 4: Extrapolation performance: qualitative comparison of model predictions on out-of-distribution test samples. Number of labeled training samples $M$: 1. Batch size: 1.
• Figure 5: Qualitative comparison of model predictions for the Helmholtz equation. Top three rows: interpolation performance for test samples from the training distributions ($a(x,y)$ composed of Gaussians). Bottom three rows: extrapolation performance for out-of-distribution test samples ($a(x,y)$ composed of wavy stripes). Number of labeled training samples $M$: 2048. Batch size: 40.