Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Crystalformer: Infinitely Connected Attention for Periodic Structure Encoding

Tatsunori Taniai, Ryo Igarashi, Yuta Suzuki, Naoya Chiba, Kotaro Saito, Yoshitaka Ushiku, Kanta Ono

TL;DR

This work introduces Crystalformer, a Transformer-based encoder tailored for periodic crystal structures. By interpreting infinite interatomic interactions as neural potential summation, the model employs distance-decay attention with explicit periodic spatial and edge encodings, yielding a pseudo-finite, tractable attention mechanism. Crystalformer achieves state-of-the-art or competitive performance on Materials Project and JARVIS-DFT regression tasks while using substantially fewer parameters than prior Transformer-based approaches, and it benefits from a normalization-free design and stochastic weight averaging. The approach offers a scalable, physics-informed pathway to capture long-range interactions in crystals, with potential extensions to reciprocal (Fourier) space for even more efficient long-range modeling and possible inclusion of angular or higher-order terms.

Abstract

Predicting physical properties of materials from their crystal structures is a fundamental problem in materials science. In peripheral areas such as the prediction of molecular properties, fully connected attention networks have been shown to be successful. However, unlike these finite atom arrangements, crystal structures are infinitely repeating, periodic arrangements of atoms, whose fully connected attention results in infinitely connected attention. In this work, we show that this infinitely connected attention can lead to a computationally tractable formulation, interpreted as neural potential summation, that performs infinite interatomic potential summations in a deeply learned feature space. We then propose a simple yet effective Transformer-based encoder architecture for crystal structures called Crystalformer. Compared to an existing Transformer-based model, the proposed model requires only 29.4% of the number of parameters, with minimal modifications to the original Transformer architecture. Despite the architectural simplicity, the proposed method outperforms state-of-the-art methods for various property regression tasks on the Materials Project and JARVIS-DFT datasets.

Crystalformer: Infinitely Connected Attention for Periodic Structure Encoding

TL;DR

This work introduces Crystalformer, a Transformer-based encoder tailored for periodic crystal structures. By interpreting infinite interatomic interactions as neural potential summation, the model employs distance-decay attention with explicit periodic spatial and edge encodings, yielding a pseudo-finite, tractable attention mechanism. Crystalformer achieves state-of-the-art or competitive performance on Materials Project and JARVIS-DFT regression tasks while using substantially fewer parameters than prior Transformer-based approaches, and it benefits from a normalization-free design and stochastic weight averaging. The approach offers a scalable, physics-informed pathway to capture long-range interactions in crystals, with potential extensions to reciprocal (Fourier) space for even more efficient long-range modeling and possible inclusion of angular or higher-order terms.

Abstract

Predicting physical properties of materials from their crystal structures is a fundamental problem in materials science. In peripheral areas such as the prediction of molecular properties, fully connected attention networks have been shown to be successful. However, unlike these finite atom arrangements, crystal structures are infinitely repeating, periodic arrangements of atoms, whose fully connected attention results in infinitely connected attention. In this work, we show that this infinitely connected attention can lead to a computationally tractable formulation, interpreted as neural potential summation, that performs infinite interatomic potential summations in a deeply learned feature space. We then propose a simple yet effective Transformer-based encoder architecture for crystal structures called Crystalformer. Compared to an existing Transformer-based model, the proposed model requires only 29.4% of the number of parameters, with minimal modifications to the original Transformer architecture. Despite the architectural simplicity, the proposed method outperforms state-of-the-art methods for various property regression tasks on the Materials Project and JARVIS-DFT datasets.
Paper Structure (39 sections, 50 equations, 4 figures, 6 tables)

This paper contains 39 sections, 50 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Crystal structures
  4. Self-attention with relative positions
  5. Crystalformer
  6. Infinitely connected attention as neural potential summation
  7. Network architecture
  8. Related work
  9. Experiments
  10. Training settings
  11. Crystal property prediction
  12. Model efficiency comparison
  13. Ablation studies
  14. Limitations and discussion
  15. Conclusions
  16. ...and 24 more sections

Figures (4)

  • Figure 1: 2D diagrams of crystal structure and infinitely connected attention.
  • Figure 2: Pseudo-finite periodic attention with periodic spatial and edge encoding in a matrix-tensor diagram. Scalar $\alpha_{ij}$ and vector $\bm{\beta}_{ij}$ integrate the spatial relations between unit-cell atom $i$ and the $j$'s all repeated atoms, allowing the infinitely connected attention to be performed as standard fully connected attention for finite unit-cell atoms. (Unlike usual notation, $X,Q,K,V,Y$ here denote column-vector-based feature matrices for better consistency with the notation in the main text.)
  • Figure 3: Network architecture of Crystalformer.
  • Figure A1: Energy above hull. Image courtesy of Ma, J., Hegde, V., Munira, K., Xie, Y., Keshavarz, S., Mildebrath, D., Wolverton, C., Ghosh, A., & Butler, W. (2017). Computational investigation of half-Heusler compounds for spintronics applications. Phys. Rev. B, 95, 024411.