Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Physical Simulation with Message Passing Transformer

Zeyi Xu, Yifei Li

TL;DR

A new universal architecture based on Graph Neural Network, the Message Passing Transformer, which incorporates a Message Passing framework, employs an Encoder-Processor-Decoder structure, and applies Graph Fourier Loss as loss function for model optimization is proposed.

Abstract

Machine learning methods for physical simulation have achieved significant success in recent years. We propose a new universal architecture based on Graph Neural Network, the Message Passing Transformer, which incorporates a Message Passing framework, employs an Encoder-Processor-Decoder structure, and applies Graph Fourier Loss as loss function for model optimization. To take advantage of the past message passing state information, we propose Hadamard-Product Attention to update the node attribute in the Processor, Hadamard-Product Attention is a variant of Dot-Product Attention that focuses on more fine-grained semantics and emphasizes on assigning attention weights over each feature dimension rather than each position in the sequence relative to others. We further introduce Graph Fourier Loss (GFL) to balance high-energy and low-energy components. To improve time performance, we precompute the graph's Laplacian eigenvectors before the training process. Our architecture achieves significant accuracy improvements in long-term rollouts for both Lagrangian and Eulerian dynamical systems over current methods.

Learning Physical Simulation with Message Passing Transformer

TL;DR

A new universal architecture based on Graph Neural Network, the Message Passing Transformer, which incorporates a Message Passing framework, employs an Encoder-Processor-Decoder structure, and applies Graph Fourier Loss as loss function for model optimization is proposed.

Abstract

Machine learning methods for physical simulation have achieved significant success in recent years. We propose a new universal architecture based on Graph Neural Network, the Message Passing Transformer, which incorporates a Message Passing framework, employs an Encoder-Processor-Decoder structure, and applies Graph Fourier Loss as loss function for model optimization. To take advantage of the past message passing state information, we propose Hadamard-Product Attention to update the node attribute in the Processor, Hadamard-Product Attention is a variant of Dot-Product Attention that focuses on more fine-grained semantics and emphasizes on assigning attention weights over each feature dimension rather than each position in the sequence relative to others. We further introduce Graph Fourier Loss (GFL) to balance high-energy and low-energy components. To improve time performance, we precompute the graph's Laplacian eigenvectors before the training process. Our architecture achieves significant accuracy improvements in long-term rollouts for both Lagrangian and Eulerian dynamical systems over current methods.
Paper Structure (42 sections, 18 equations, 8 figures, 5 tables)

This paper contains 42 sections, 18 equations, 8 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Approaches in Physical Systems Simulation
  4. Advancements in GNN Architectures for Simulation
  5. Bridging Graph Theory and Signal Processing
  6. Problem Formulation and Preliminaries
  7. Problem Formulation
  8. Graph Fourier Transform
  9. Mathematical Definitions
  10. Method
  11. Message Passing Transformer
  12. Encoder
  13. Processor
  14. Decoder
  15. Hadamard-Product Attention
  16. ...and 27 more sections

Figures (8)

  • Figure 1: Model Architecture of the Message Passing Transformer, visualizing the information processing procedure for the first of four Message Passing ($k=0, M=4$) times. The encoder module transposes inputs into a latent space and the decoder predicts future states by extrapolating these encoded representations. The processor unit conducts numerous iterations, each treated as a regression problem, to refine node and edge attributes.
  • Figure 2: Comparison of RMSE of velocity norm between the Lagrangian system FlagSimple and the Eulerian system CylinderFlow using our MPT model and the MGN model MGN.
  • Figure 3: (a) Comparison of Dot-Product Attention and Hadamard-Product Attention in CylinderFlow. HPA demonstrates a lower average RMSE across all rollout steps compared to the Dot-Product Attention. (b) Comparison of learnable and manual $\lambda$ settings in FlagSimple. Learnable $\lambda$ achieves lower error compared to manual settings. (c) The impact of varying segmentation rates $s_r$ in FlagSimple. Different segmentation rates do not significantly impact the final results.
  • Figure 4: Flag Simple
  • Figure 5: Deforming Plate
  • ...and 3 more figures