Physics-Guided Transformer (PGT): Physics-Aware Attention Mechanism for PINNs

Abstract

Reconstructing continuous physical fields from sparse, irregular observations is a central challenge in scientific machine learning, particularly for systems governed by partial differential equations (PDEs). Existing physics-informed methods typically enforce governing equations as soft penalty terms during optimization, often leading to gradient imbalance, instability, and degraded physical consistency under limited data. We introduce the Physics-Guided Transformer (PGT), a neural architecture that embeds physical structure directly into the self-attention mechanism. Specifically, PGT incorporates a heat-kernel-derived additive bias into attention logits, encoding diffusion dynamics and temporal causality within the representation. Query coordinates attend to these physics-conditioned context tokens, and the resulting features are decoded using a FiLM-modulated sinusoidal implicit network that adaptively controls spectral response. We evaluate PGT on the one-dimensional heat equation and two-dimensional incompressible Navier-Stokes systems. In sparse 1D reconstruction with 100 observations, PGT achieves a relative L2 error of 5.9e-3, significantly outperforming both PINNs and sinusoidal representations. In the 2D cylinder wake problem, PGT uniquely achieves both low PDE residual (8.3e-4) and competitive relative error (0.034), outperforming methods that optimize only one objective. These results demonstrate that embedding physics within attention improves stability, generalization, and physical fidelity under data-scarce conditions.

Figures (6)

• Figure 1: Overview of the PGT architecture. The physics-guided Transformer encoder processes sparse observations into latent context tokens, which are then used to condition the FiLM-modulated SIREN decoder for continuous field reconstruction.
• Figure 2: Training error convergence (relative $L^2$) for PINN, SIREN, and PGT on the 1D heat diffusion sparse reconstruction task ($M=100$ observations). PGT exhibits sustained monotonic decay, whereas PINN and SIREN plateau at much higher error levels.
• Figure 3: (a) Error components breakdown. (b) Error components contribution analysis.
• Figure 4: Qualitative comparison between ground truth and PGT reconstruction for the 2D Navier--Stokes problem. Rows correspond to $u$, $v$, and $p$ fields. Columns show ground truth, PGT prediction, and absolute error.
• Figure 5: Training error convergence for PGT and baseline methods on the 2D Navier--Stokes problem.