Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks

Zhiyuan Zhao, Xueying Ding, B. Aditya Prakash

TL;DR

PINNs struggle to propagate initial conditions in time-aware PDEs due to neglect of temporal dependencies. The authors propose PINNsFormer, a Transformer-based framework that converts pointwise inputs into pseudo time sequences via a Pseudo Sequence Generator, and uses a Spatio-Temporal Mixer with an Encoder-Decoder and a novel Wavelet activation to capture temporal dynamics. Empirical results show PINNsFormer outperforms traditional PINNs and variants on convection, 1D-reaction, 1D-wave, and 2D Navier–Stokes PDEs, with smoother loss landscapes and better generalization, even when combined with NTK learning schemes. The approach balances accuracy and generalization with manageable overhead, and the Wavelet activation suggests broader applicability beyond PINNs.

Abstract

Physics-Informed Neural Networks (PINNs) have emerged as a promising deep learning framework for approximating numerical solutions to partial differential equations (PDEs). However, conventional PINNs, relying on multilayer perceptrons (MLP), neglect the crucial temporal dependencies inherent in practical physics systems and thus fail to propagate the initial condition constraints globally and accurately capture the true solutions under various scenarios. In this paper, we introduce a novel Transformer-based framework, termed PINNsFormer, designed to address this limitation. PINNsFormer can accurately approximate PDE solutions by utilizing multi-head attention mechanisms to capture temporal dependencies. PINNsFormer transforms point-wise inputs into pseudo sequences and replaces point-wise PINNs loss with a sequential loss. Additionally, it incorporates a novel activation function, Wavelet, which anticipates Fourier decomposition through deep neural networks. Empirical results demonstrate that PINNsFormer achieves superior generalization ability and accuracy across various scenarios, including PINNs failure modes and high-dimensional PDEs. Moreover, PINNsFormer offers flexibility in integrating existing learning schemes for PINNs, further enhancing its performance.

PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks

TL;DR

PINNs struggle to propagate initial conditions in time-aware PDEs due to neglect of temporal dependencies. The authors propose PINNsFormer, a Transformer-based framework that converts pointwise inputs into pseudo time sequences via a Pseudo Sequence Generator, and uses a Spatio-Temporal Mixer with an Encoder-Decoder and a novel Wavelet activation to capture temporal dynamics. Empirical results show PINNsFormer outperforms traditional PINNs and variants on convection, 1D-reaction, 1D-wave, and 2D Navier–Stokes PDEs, with smoother loss landscapes and better generalization, even when combined with NTK learning schemes. The approach balances accuracy and generalization with manageable overhead, and the Wavelet activation suggests broader applicability beyond PINNs.

Abstract

Physics-Informed Neural Networks (PINNs) have emerged as a promising deep learning framework for approximating numerical solutions to partial differential equations (PDEs). However, conventional PINNs, relying on multilayer perceptrons (MLP), neglect the crucial temporal dependencies inherent in practical physics systems and thus fail to propagate the initial condition constraints globally and accurately capture the true solutions under various scenarios. In this paper, we introduce a novel Transformer-based framework, termed PINNsFormer, designed to address this limitation. PINNsFormer can accurately approximate PDE solutions by utilizing multi-head attention mechanisms to capture temporal dependencies. PINNsFormer transforms point-wise inputs into pseudo sequences and replaces point-wise PINNs loss with a sequential loss. Additionally, it incorporates a novel activation function, Wavelet, which anticipates Fourier decomposition through deep neural networks. Empirical results demonstrate that PINNsFormer achieves superior generalization ability and accuracy across various scenarios, including PINNs failure modes and high-dimensional PDEs. Moreover, PINNsFormer offers flexibility in integrating existing learning schemes for PINNs, further enhancing its performance.
Paper Structure (17 sections, 1 theorem, 15 equations, 9 figures, 7 tables)

This paper contains 17 sections, 1 theorem, 15 equations, 9 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Pseudo Sequence Generator
  5. Model Architecture
  6. Wavelet Activation
  7. Learning Scheme
  8. Loss Landscape Analysis
  9. Experiments
  10. Setup
  11. Mitigating Failure Modes of PINNs
  12. Flexibility in Incorporating Variant Learning Schemes
  13. Generalization on High-Dimensional PDEs
  14. Conclusion
  15. Appendix A: Model Hyperparameters
  16. ...and 2 more sections

Key Result

Proposition 1

Let $\mathcal{N}$ be a two-hidden-layer neural network with infinite width, equipped with Wavelet activation function, then $\mathcal{N}$ is a universal approximator for any real-valued target f.

Figures (9)

  • Figure 1: Architecture of proposed PINNsFormer. PINNsFormer generates a pseudo sequence based on pointwise input features. It outputs the corresponding sequential approximated solution. The first approximation of the sequence is the desired solution $\hat{u}(\boldsymbol{x},t)$.
  • Figure 2: The architecture of PINNsFormer's Encoder-Decoder Layers. The decoder is not equipped with self-attentions.
  • Figure 3: Visualization of the loss landscape for PINNs (left) and PINNsFormer (right) on a logarithmic scale. The loss landscape of PINNsFormer is significantly smoother than conventional PINNs.
  • Figure 4: Prediction (left) and absolute error (right) of PINNs (up) and PINNsFormer (bottom) on convection equation. PINNsFormer shows success in mitigating the failure mode than PINNs.
  • Figure 5: Training loss vs. Iterations of PINNs and PINNsFormer on the Navier-Stokes equation.
  • ...and 4 more figures

Theorems & Definitions (1)

  • Proposition 1