Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HAMLET: Graph Transformer Neural Operator for Partial Differential Equations

Andrey Bryutkin, Jiahao Huang, Zhongying Deng, Guang Yang, Carola-Bibiane Schönlieb, Angelica Aviles-Rivero

Abstract

We present a novel graph transformer framework, HAMLET, designed to address the challenges in solving partial differential equations (PDEs) using neural networks. The framework uses graph transformers with modular input encoders to directly incorporate differential equation information into the solution process. This modularity enhances parameter correspondence control, making HAMLET adaptable to PDEs of arbitrary geometries and varied input formats. Notably, HAMLET scales effectively with increasing data complexity and noise, showcasing its robustness. HAMLET is not just tailored to a single type of physical simulation, but can be applied across various domains. Moreover, it boosts model resilience and performance, especially in scenarios with limited data. We demonstrate, through extensive experiments, that our framework is capable of outperforming current techniques for PDEs.

HAMLET: Graph Transformer Neural Operator for Partial Differential Equations

Abstract

We present a novel graph transformer framework, HAMLET, designed to address the challenges in solving partial differential equations (PDEs) using neural networks. The framework uses graph transformers with modular input encoders to directly incorporate differential equation information into the solution process. This modularity enhances parameter correspondence control, making HAMLET adaptable to PDEs of arbitrary geometries and varied input formats. Notably, HAMLET scales effectively with increasing data complexity and noise, showcasing its robustness. HAMLET is not just tailored to a single type of physical simulation, but can be applied across various domains. Moreover, it boosts model resilience and performance, especially in scenarios with limited data. We demonstrate, through extensive experiments, that our framework is capable of outperforming current techniques for PDEs.
Paper Structure (14 sections, 1 theorem, 26 equations, 7 figures, 6 tables)

This paper contains 14 sections, 1 theorem, 26 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology: HAMLET
  4. Graph Construction for HAMLET
  5. Graph Transformer for HAMLET
  6. Unboxing the HAMLET Architecture
  7. Experimental Results
  8. Dataset & Implementation Details
  9. Results & Discussion
  10. Conclusion
  11. Further Numerical & Visual Results.
  12. Discretisation Invariance & Further Definitions.
  13. Further Details on the Datasets & PDEs.
  14. Further Details on Model Architecture & Implementations

Key Result

Proposition 3.1

The residual block of the graph transformer layer, as proposed above, can be seen as a special case of the integration kernel of the neural operator.

Figures (7)

  • Figure 1: Architectural overview of HAMLET. The diagram depicts the transformation of input observations into graph representations, followed by node feature embedding using a multilayer perceptron (MLP) to prepare key, value and query vectors for the graph transformer. The transformer captures data structure and node relationships, which are then refined by the cross-former with query positions. Finally, shared MLPs propagate the embeddings to model system behaviour over time.
  • Figure 2: Performance impacts of graph construction and position encoding on model accuracy. (A) The trade-off between circular graph radius and nRMSE alongside edge count. (B) Comparison of nRMSE across position encoding methods.
  • Figure 3: Comparison of model predictions ($128\times128$) against a reference for two input resolutions ($32\times32$ and $64\times64$). Left: Heatmap comparisons of the reference and predicted outcomes by OFormer and HAMLET models. Right: Corresponding nRMSE values, demonstrating the superior accuracy of HAMLET.
  • Figure 4: Predictions and corresponding error maps for Geo-FNO, OFormer, and HAMLET models using Darcy Flow ($\beta=1.0$).
  • Figure 5: Time evolution (activator) of the diffusion-reaction process as predicted by OFormer and HAMLET compared to the reference solution. HAMLET 's predictions closely match the reference, showcasing its efficacy in dynamic PDE modelling.
  • ...and 2 more figures

Theorems & Definitions (3)

  • Proposition 3.1
  • proof
  • Definition 2.1: Neural operator $\tilde{\mathcal{S}}_\mu$