Benchmarking Graphormer on Large-Scale Molecular Modeling Datasets

TL;DR

This work benchmarks Graphormer for large-scale molecular modeling, introducing architectural refinements and a 3D extension that leverage a Transformer backbone with graph-specific encodings. It demonstrates that Post-LN layer normalization improves generalization on PCQM4M, and that 3D spatial encodings with a rotationally equivariant 3D attention layer enhance performance on OC20’s catalyst-adsorbate tasks. The authors provide a theoretical lens grounded in distributed computing to explain why a global receptive field broadens expressiveness beyond traditional local MPGNNs. The results, along with publicly available code, offer stronger baselines for 2D and 3D molecular modeling and insights into non-local attention for chemical prediction tasks.

Abstract

This technical note describes the recent updates of Graphormer, including architecture design modifications, and the adaption to 3D molecular dynamics simulation. With these simple modifications, Graphormer could attain better results on large-scale molecular modeling datasets than the vanilla one, and the performance gain could be consistently obtained on 2D and 3D molecular graph modeling tasks. In addition, we show that with a global receptive field and an adaptive aggregation strategy, Graphormer is more powerful than classic message-passing-based GNNs. Empirically, Graphormer could achieve much less MAE than the originally reported results on the PCQM4M quantum chemistry dataset used in KDD Cup 2021. In the meanwhile, it greatly outperforms the competitors in the recent Open Catalyst Challenge, which is a competition track on NeurIPS 2021 workshop, and aims to model the catalyst-adsorbate reaction system with advanced AI models. All codes could be found at https://github.com/Microsoft/Graphormer.

Theorems & Definitions (6)

• Theorem 1.1
• Theorem 1.2
• Theorem 1.3
• Theorem 1.4
• Theorem 1.5
• Theorem 1.6