Fine-Tuned Language Models Generate Stable Inorganic Materials as Text

TL;DR

<p>We address the problem of generating stable inorganic materials by fine-tuning large language models on text-encoded crystal structures. Our approach uses parameter-efficient LoRA adapters on a pre-trained LLaMA-2 model to enable unconditional, text-conditioned, and infilling generation, leveraging simple string representations and targeted augmentations. The strongest model (LLaMA-2 70B) achieves metastable material generation at about 49% versus 28% for a diffusion baseline (CDVAE), with stability assessed via $E_{ ext{hull}}$ predictions from M3GNet and DFT. The work demonstrates that scale improves symmetry learning and that LLMs can serve as flexible, fast, multitask generators for atomistic design, with practical implications for materials discovery and design workflows.</p>

Abstract

We propose fine-tuning large language models for generation of stable materials. While unorthodox, fine-tuning large language models on text-encoded atomistic data is simple to implement yet reliable, with around 90% of sampled structures obeying physical constraints on atom positions and charges. Using energy above hull calculations from both learned ML potentials and gold-standard DFT calculations, we show that our strongest model (fine-tuned LLaMA-2 70B) can generate materials predicted to be metastable at about twice the rate (49% vs 28%) of CDVAE, a competing diffusion model. Because of text prompting's inherent flexibility, our models can simultaneously be used for unconditional generation of stable material, infilling of partial structures and text-conditional generation. Finally, we show that language models' ability to capture key symmetries of crystal structures improves with model scale, suggesting that the biases of pretrained LLMs are surprisingly well-suited for atomistic data.

Figures (8)

• Figure 1: Overview of our approach to materials generation with large language models. Using string formatted crystals and task-specific prompting, we enable unconditional stable materials generation, text-condition materials generation, and structural infilling. Base LLaMA-2 models are fine-tuned on a database of known inorganic materials Liu2020MultilingualDP using low-rank adapters.
• Figure 2: (left) We convert the crystal lattice, atom identities, and atom positions into strings. The model is trained to generate a structures conditioned on the text prompt, which might contain additional information about the composition, properties, or a starting structure to modify. (right) Energy above hull ($E_{\text{hull}}$) quantifies the stability of a material. A crystal with $E_{\text{hull}} < 0.1$ will be energetically favorable both in its structure and composition.
• Figure 3: A sample with "hallucinated" element identities (Ln).
• Figure 4: Stability of LLaMA samples compared to CDVAE xie2021crystal. Fine-tuned LLaMA-2 70B generates a higher rate of metastable ($\hat{E}_{\text{hull}} <$ 0.1) and stable materials than CDVAE, using estimates of $\hat{E}_{\text{hull}}$ from both M3GNet chen2022universal and VASP hafner2008ab. Because of computational cost, we only run VASP on structures predicted to be stable by M3GNet. Stable materials generated by LLaMA are also more diverse (as quantified by Matminer featurization ward2018matminer) than stable samples from CDVAE. We include sampled stable structures, shown as (2,2,2) supercells, which display a high-degree of regularity and understanding of three-dimensional space.
• Figure 5: Translation invariance on test data and ability to generate stable materials increase in proportion. Larger models learn invariances from augmentations more effectively during training, likely as a result of their preference for abstract and compressible patterns.