Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Universal Neural Operators through Multiphysics Pretraining

Mikhail Masliaev, Dmitry Gusarov, Ilya Markov, Alexander Hvatov

TL;DR

This work addresses the high cost of training neural operators for PDE-based simulations by proposing a multiphysics pretraining framework. It combines transformer- and state-space–based neural operators with adapter-based transfer learning to enable cross-domain generalization across varying inputs and physics. The main contributions include a modular pretraining/fine-tuning scheme, demonstrations on multiple PDEs (e.g., Burgers', Gray-Scott, Navier–Stokes), and evidence that pretraining on heterogeneous multiphysics data improves generalization while reducing fine-tuning costs. This approach pushes toward a foundational PDE dynamics model with practical benefits for rapid deployment in new multiphysics scenarios and lays groundwork for future data augmentation and symmetry-based enhancements.

Abstract

Although neural operators are widely used in data-driven physical simulations, their training remains computationally expensive. Recent advances address this issue via downstream learning, where a model pretrained on simpler problems is fine-tuned on more complex ones. In this research, we investigate transformer-based neural operators, which have previously been applied only to specific problems, in a more general transfer learning setting. We evaluate their performance across diverse PDE problems, including extrapolation to unseen parameters, incorporation of new variables, and transfer from multi-equation datasets. Our results demonstrate that advanced neural operator architectures can effectively transfer knowledge across PDE problems.

Towards Universal Neural Operators through Multiphysics Pretraining

TL;DR

This work addresses the high cost of training neural operators for PDE-based simulations by proposing a multiphysics pretraining framework. It combines transformer- and state-space–based neural operators with adapter-based transfer learning to enable cross-domain generalization across varying inputs and physics. The main contributions include a modular pretraining/fine-tuning scheme, demonstrations on multiple PDEs (e.g., Burgers', Gray-Scott, Navier–Stokes), and evidence that pretraining on heterogeneous multiphysics data improves generalization while reducing fine-tuning costs. This approach pushes toward a foundational PDE dynamics model with practical benefits for rapid deployment in new multiphysics scenarios and lays groundwork for future data augmentation and symmetry-based enhancements.

Abstract

Although neural operators are widely used in data-driven physical simulations, their training remains computationally expensive. Recent advances address this issue via downstream learning, where a model pretrained on simpler problems is fine-tuned on more complex ones. In this research, we investigate transformer-based neural operators, which have previously been applied only to specific problems, in a more general transfer learning setting. We evaluate their performance across diverse PDE problems, including extrapolation to unseen parameters, incorporation of new variables, and transfer from multi-equation datasets. Our results demonstrate that advanced neural operator architectures can effectively transfer knowledge across PDE problems.

Paper Structure

This paper contains 10 sections, 3 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Neural operator learning:
  5. Improving the generalization ability of neural operators:
  6. Pre-training and fine-tuning:
  7. Experiments
  8. Out-of-sample parameter values scenario
  9. Input function set extension scenario & General multi-physics learning
  10. Conclusion

Figures (1)

  • Figure 1: Scheme of the neural operator pre-training and fine-tuning stages. In this scheme, FNO blocks denote arbitrary kernel integral operators, including the transformer-based architectures. The separate physics $1$ to $N$ may vary from the different manifestations of the same system to multiple different physics (but with the same problem dimensionality) in the dataset collection.