AtomBench: A Benchmark for Generative Atomic Structure Models using GPT, Diffusion, and Flow Architectures

TL;DR

The paper tackles the lack of standardized benchmarks for inverse-design crystal-structure generation in superconductors by comparing three architectures—AtomGPT (transformer), CDVAE (diffusion VAE), and FlowMM (Riemannian flow matching)—on two DF-relaxed datasets. It uses reconstruction-focused metrics: mean absolute error (MAE) on lattice parameters, Kullback–Leibler Divergence $D_{KL}(P\|Q)$ between predicted and ground-truth lattice-parameter distributions, and RMSE on atomic coordinates, with crystal description given by six lattice parameters $\\{a,b,c,\\alpha,\\beta,\\gamma\\}$ and fractional coordinates $\\mathbf{r}_i$ under Niggli reduction. Key findings show CDVAE most accurately reconstructs lattice parameters (lowest $D_{KL}$ and MAE) across datasets, AtomGPT bests FlowMM on atomic-coordinate RMSE, and FlowMM generally lags in lattice reconstruction; these differences are linked to the amount of conditioning information each model receives ($\ ext{A}$, $T_c$, $X$, $L$). The study highlights the need to equalize prior information across models to isolate architectural biases and informs how to benchmark inverse-design methods for accelerating materials discovery.

Abstract

Generative models have become significant assets in the exploration and identification of new materials, enabling the rapid proposal of candidate crystal structures that satisfy target properties. Despite the increasing adoption of diverse architectures, a rigorous comparative evaluation of their performance on materials datasets is lacking. In this work, we present a systematic benchmark of three representative generative models- AtomGPT (a transformer-based model), Crystal Diffusion Variational Autoencoder (CDVAE), and FlowMM (a Riemannian flow matching model). These models were trained to reconstruct crystal structures from subsets of two publicly available superconductivity datasets- JARVIS Supercon 3D and DS A/B from the Alexandria database. Performance was assessed using the Kullback-Leibler (KL) divergence between predicted and reference distributions of lattice parameters, as well as the mean absolute error (MAE) of individual lattice constants. For the computed KLD and MAE scores, CDVAE performs most favorably, followed by AtomGPT, and then FlowMM. All benchmarking code and model configurations will be made publicly available at https://github.com/atomgptlab/atombench_inverse.

Figures (8)

• Figure 1: Diagram showing the inverse model benchmarking study design. We compare three generative AI inverse models-AtomGPTChoudhary2023AtomGPT, a language model; CDVAEXie2022CDVAE, a diffusion variational autoencoder; and FlowMMMiller2024FlowMM, a flow-matching network-each seperately trained on two superconductivity DFT datasets (JARVIS Supercon-3DChoudhary_2022 and Alexandria DS-A/BCerqueiraHydrideSupercons) for a total of six benchmarks. Ten percent of each dataset is held out to test reconstruction performance, which we statistically quantify before submitting these results to the JARVIS-Leaderboard.
• Figure 2: Statistical comparison of the JARVIS Supercon-3D and Alexandria DS-A/B superconductivity datasets. (a,b) Pie charts show elemental compositions for the 23 most represented elements. In JARVIS, oxygen, aluminum, and titanium are most common (9.3%, 5.0%, and 4.8%), while in Alexandria, titanium, rhodium, and aluminum dominate (6.6%, 5.0%, and 5.1%). (c) The overlain histogram compares superconducting critical temperature ($T_c$) distributions, both decaying exponentially with $T_c$, though Supercon-3D contains a larger fraction of high-$T_c$ materials. (d) The overlain bar chart shows crystal system distributions: DS-A/B is dominated by cubic and tetragonal phases ($>$80%), whereas Supercon-3D exhibits greater diversity, with $\sim$45% cubic and a broader spread across all seven systems.
• Figure 3: Reconstruction performance of AtomGPT, CDVAE, and FlowMM on the Alexandria DS-A/B test set (825 structures) for three representative Niggli-reduced lattice parameters, $a$, $c$, and $\gamma$. The blue distributions are the target distributions directly obtained from the dataset, and the gold distributions are the predictions made by the models with the goal of matching the target distributions. CDVAE appears to match the target distribution most closely, followed by AtomGPT, and with FlowMM's predictions matching the target the least closely.
• Figure 4: Reconstruction performance of AtomGPT, CDVAE, and FlowMM on the JARVIS Supercon-3D test set (105 structures) for three representative Niggli-reduced lattice parameters, $a$, $c$, and $\gamma$. The blue distributions are the target distributions directly obtained from the dataset, and the gold distributions are the predictions made by the models with the goal of matching the target distributions. On the average, CDVAE has the lowest KLD, followed by AtomGPT and then FlowMM. It is ambiguous which model's predictions visually appear to follow the target distributions most closely, reasonably due to the lower volume of test structures.
• Figure 5: Kullback-Leibler Divergence in units of nats between the predicted and target distributions for all six Niggli-reduced lattice parameters ($a$, $b$, $c$, $\alpha$, $\beta$, $\gamma$) for a total of six experiments using three models and two datasets. CDVAE has the most favorable KLD scores for both datasets, followed by AtomGPT and then FlowMM.