Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Latent Neural Operator Pretraining for Solving Time-Dependent PDEs

Tian Wang, Chuang Wang

TL;DR

This work proposes the Latent Neural Operator Pretraining (LNOP) framework based on the Latent Neural Operator (LNO) backbone and achieves universal transformation through pretraining on hybrid time-dependent PDE dataset to extract representations of different physical systems and solve various time-dependent PDEs in the latent space through finetuning on single PDE dataset.

Abstract

Pretraining methods gain increasing attraction recently for solving PDEs with neural operators. It alleviates the data scarcity problem encountered by neural operator learning when solving single PDE via training on large-scale datasets consisting of various PDEs and utilizing shared patterns among different PDEs to improve the solution precision. In this work, we propose the Latent Neural Operator Pretraining (LNOP) framework based on the Latent Neural Operator (LNO) backbone. We achieve universal transformation through pretraining on hybrid time-dependent PDE dataset to extract representations of different physical systems and solve various time-dependent PDEs in the latent space through finetuning on single PDE dataset. Our proposed LNOP framework reduces the solution error by 31.7% on four problems and can be further improved to 57.1% after finetuning. On out-of-distribution dataset, our LNOP model achieves roughly 50% lower error and 3$\times$ data efficiency on average across different dataset sizes. These results show that our method is more competitive in terms of solution precision, transfer capability and data efficiency compared to non-pretrained neural operators.

Latent Neural Operator Pretraining for Solving Time-Dependent PDEs

TL;DR

This work proposes the Latent Neural Operator Pretraining (LNOP) framework based on the Latent Neural Operator (LNO) backbone and achieves universal transformation through pretraining on hybrid time-dependent PDE dataset to extract representations of different physical systems and solve various time-dependent PDEs in the latent space through finetuning on single PDE dataset.

Abstract

Pretraining methods gain increasing attraction recently for solving PDEs with neural operators. It alleviates the data scarcity problem encountered by neural operator learning when solving single PDE via training on large-scale datasets consisting of various PDEs and utilizing shared patterns among different PDEs to improve the solution precision. In this work, we propose the Latent Neural Operator Pretraining (LNOP) framework based on the Latent Neural Operator (LNO) backbone. We achieve universal transformation through pretraining on hybrid time-dependent PDE dataset to extract representations of different physical systems and solve various time-dependent PDEs in the latent space through finetuning on single PDE dataset. Our proposed LNOP framework reduces the solution error by 31.7% on four problems and can be further improved to 57.1% after finetuning. On out-of-distribution dataset, our LNOP model achieves roughly 50% lower error and 3 data efficiency on average across different dataset sizes. These results show that our method is more competitive in terms of solution precision, transfer capability and data efficiency compared to non-pretrained neural operators.

Paper Structure

This paper contains 24 sections, 7 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Operator
  4. PDE Pretraining
  5. Method
  6. Problem Setup
  7. Latent Neural Operator
  8. Frame Design
  9. Overall
  10. Pretraining PhCA Encoder/Decoder
  11. Finetuning Propagator
  12. Dataset
  13. Navier-Stokes Equation
  14. Shallow-Water Equation
  15. Burgers' Equation
  16. ...and 9 more sections

Figures (2)

  • Figure 1: The overall architecture of Latent Neural Operator Pretraining. We pretrain the Latent Neural Operator (LNO) on datasets containing various time-dependent PDE problems. This enables the PhCA encoder and decoder within the LNO to learn general transformation, which is used to extract PDE representations in the latent space. Subsequently, we finetune the LNO on out-of-distribution downstream tasks to apply the learned universal transformation.
  • Figure 2: Results of scaling experiments. (a) Impact of representation token dimension on solution precision of various PDE problems. (b) Impact of representation token quantity on solution precision of various PDE problems.