Generative Hierarchical Materials Search

TL;DR

Experiments show that GenMS outperforms other alternatives of directly using language models to generate structures both in satisfying user request and in generating low-energy structures, and can form the foundation for more complex structure generation in near future.

Abstract

Generative models trained at scale can now produce text, video, and more recently, scientific data such as crystal structures. In applications of generative approaches to materials science, and in particular to crystal structures, the guidance from the domain expert in the form of high-level instructions can be essential for an automated system to output candidate crystals that are viable for downstream research. In this work, we formulate end-to-end language-to-structure generation as a multi-objective optimization problem, and propose Generative Hierarchical Materials Search (GenMS) for controllable generation of crystal structures. GenMS consists of (1) a language model that takes high-level natural language as input and generates intermediate textual information about a crystal (e.g., chemical formulae), and (2) a diffusion model that takes intermediate information as input and generates low-level continuous value crystal structures. GenMS additionally uses a graph neural network to predict properties (e.g., formation energy) from the generated crystal structures. During inference, GenMS leverages all three components to conduct a forward tree search over the space of possible structures. Experiments show that GenMS outperforms other alternatives of directly using language models to generate structures both in satisfying user request and in generating low-energy structures. We confirm that GenMS is able to generate common crystal structures such as double perovskites, or spinels, solely from natural language input, and hence can form the foundation for more complex structure generation in near future.

Figures (5)

• Figure 1: Overview of GenMS. GenMS takes a high-level language instruction as input, retrieves relevant information from the internet, and samples from a high-level LLM ($\pi_\text{hi}$) to generate candidate formulae that satisfy user requirement. GenMS then samples from a low-level diffusion model ($\pi_\text{lo}$) to generate structures conditioned on candidate formulae. Sampled structures then go through a property prediction module for selection.
• Figure 2: Diffusion architecture with compact crystal representation. The diffusion model in GenMS represents crystal structures by the $x,y,z$ location of each atom plus the atom number $a$ represented as a continuous value. Each atom undergoes blocks consisting of multi-layer perceptrons followed by order-invariant self-attention. The MLP and self-attention blocks are repeated $k$ times where each repetition increases the dimension of the hidden units. The concatenation of skip connections are employed as in other U-Net architectures.
• Figure 3: Qualitative evaluation. We test GenMS on a set of ad hoc language inputs to generate plausible examples from well-known crystal families. GenMS is able to search for the corresponding structures that satisfy user requests and have plausible initial geometries. Visualization provided by VESTA momma2011vesta.
• Figure 4: Formation energy between Best-of-N and a single sample. Both according to energy predicted by GNN and calculated by DFT, best-of-N with N = 10 leads to improvements in energy compared to single samples for 80% of 1,000 compositions considered.
• Figure : Generative Hierarchical Materials Search