Generative Inverse Design of Crystal Structures via Diffusion Models with Transformers

TL;DR

This work introduces a Transformer-based diffusion model for generative inverse design of crystal structures, addressing the multi-variable representation of lattices, fractional coordinates, and atomic species. It implements a joint diffusion framework with diffusion on lattices, coordinates, and species, combined with a Transformer denoiser that uses relative position encoding to model inter-atomic interactions. Two conditioning strategies, Condition Dependent Initialization (CDI) and Condition Dependent Self-attention (CDS), are proposed to steer generation toward target properties, and are evaluated across three diverse datasets, showing near-ideal or improved success rates compared to prior methods, with performance depending on the dataset. The findings highlight the potential of dataset-aware conditioning and Transformer backbones for high-precision inverse design in crystalline materials, while pointing to future work in DFT validation and diversity metrics to assess novelty in designed structures.

Abstract

Recent advances in deep learning have enabled the generation of realistic data by training generative models on large datasets of text, images, and audio. While these models have demonstrated exceptional performance in generating novel and plausible data, it remains an open question whether they can effectively accelerate scientific discovery through the data generation and drive significant advancements across various scientific fields. In particular, the discovery of new inorganic materials with promising properties poses a critical challenge, both scientifically and for industrial applications. However, unlike textual or image data, materials, or more specifically crystal structures, consist of multiple types of variables - including lattice vectors, atom positions, and atomic species. This complexity in data give rise to a variety of approaches for representing and generating such data. Consequently, the design choices of generative models for crystal structures remain an open question. In this study, we explore a new type of diffusion model for the generative inverse design of crystal structures, with a backbone based on a Transformer architecture. We demonstrate our models are superior to previous methods in their versatility for generating crystal structures with desired properties. Furthermore, our empirical results suggest that the optimal conditioning methods vary depending on the dataset.

Figures (2)

• Figure 1: A conceptual diagram of a crystal structure represented in 2D for intuitive understanding. The red arrows represent the lattice vectors of the unit cell, while colored dots represent atoms, with the colors corresponding to different atomic species. The gray region highlights the area of the unit cell.
• Figure 2: (a) Overview of the base model architecture. (b) Overview of the conditional model with condition-dependent initialization method. (c) Overview of the conditional model with condition-dependent self-attention method. The models of crystal structures in this figure were visualized using VESTA momma2008.