Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FlowLLM: Flow Matching for Material Generation with Large Language Models as Base Distributions

Anuroop Sriram, Benjamin Kurt Miller, Ricky T. Q. Chen, Brandon M. Wood

TL;DR

This paper introduces FlowLLM, a novel generative model that combines large language models (LLMs) and Riemannian flow matching (RFM) to design novel crystalline materials and significantly outperforms state-of-the-art methods on a difficult problem.

Abstract

Material discovery is a critical area of research with the potential to revolutionize various fields, including carbon capture, renewable energy, and electronics. However, the immense scale of the chemical space makes it challenging to explore all possible materials experimentally. In this paper, we introduce FlowLLM, a novel generative model that combines large language models (LLMs) and Riemannian flow matching (RFM) to design novel crystalline materials. FlowLLM first fine-tunes an LLM to learn an effective base distribution of meta-stable crystals in a text representation. After converting to a graph representation, the RFM model takes samples from the LLM and iteratively refines the coordinates and lattice parameters. Our approach significantly outperforms state-of-the-art methods, increasing the generation rate of stable materials by over three times and increasing the rate for stable, unique, and novel crystals by $\sim50\%$ - a huge improvement on a difficult problem. Additionally, the crystals generated by FlowLLM are much closer to their relaxed state when compared with another leading model, significantly reducing post-hoc computational cost.

FlowLLM: Flow Matching for Material Generation with Large Language Models as Base Distributions

TL;DR

This paper introduces FlowLLM, a novel generative model that combines large language models (LLMs) and Riemannian flow matching (RFM) to design novel crystalline materials and significantly outperforms state-of-the-art methods on a difficult problem.

Abstract

Material discovery is a critical area of research with the potential to revolutionize various fields, including carbon capture, renewable energy, and electronics. However, the immense scale of the chemical space makes it challenging to explore all possible materials experimentally. In this paper, we introduce FlowLLM, a novel generative model that combines large language models (LLMs) and Riemannian flow matching (RFM) to design novel crystalline materials. FlowLLM first fine-tunes an LLM to learn an effective base distribution of meta-stable crystals in a text representation. After converting to a graph representation, the RFM model takes samples from the LLM and iteratively refines the coordinates and lattice parameters. Our approach significantly outperforms state-of-the-art methods, increasing the generation rate of stable materials by over three times and increasing the rate for stable, unique, and novel crystals by - a huge improvement on a difficult problem. Additionally, the crystals generated by FlowLLM are much closer to their relaxed state when compared with another leading model, significantly reducing post-hoc computational cost.

Paper Structure

This paper contains 45 sections, 11 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Crystal representation
  5. Equivariance & Invariance
  6. Symmetries of crystals
  7. Method
  8. Overview of training
  9. Overview of sampling
  10. Large Language Model ( ) for Crystals
  11. Tokenization
  12. Training
  13. Sampling
  14. Symmetries in LLMs
  15. Riemannian Flow Matching ( ) for Crystals
  16. ...and 30 more sections

Figures (3)

  • Figure 1: FlowLLM generative process: the fine-tuned LLM is first prompted with an unconditional query to generate an initial material representation. This material is then iteratively transformed by the RFM model to update its atom positions and lattice parameters. The atom types are static in RFM.
  • Figure 2: Left: String encoding of materials used to train the LLM based on Gruver et al.gruver2024fine. Right: An example prompt used during training. The conditioning information in blue is optional, and can be replaced with conditioning on other properties as well. The text in red is replaced with the crystal string representation shown on the left.
  • Figure 3: (a) Histogram of $E^{\text{hull}}$ values comparing FlowLLM with prior models. The dashed line shows thermodynamic stability threshold ($E^{\text{hull}}$ = 0). (b) Histogram of N-ary compared to the data distribution. (c) Structural validity as a function of number of integration steps.