PhononBench:A Large-Scale Phonon-Based Benchmark for Dynamical Stability in Crystal Generation

TL;DR

PhononBench introduces the first large-scale benchmark for dynamical stability in AI-generated crystals by combining MatterSim-driven phonon calculations with high-throughput generation from six approaches. It demonstrates that dynamical stability remains a major hurdle, identifying 28,119 phonon-stable structures and highlighting how pretraining data, diffusion architectures, and space-group constraints affect stability. The study provides a robust, open evaluation framework and dataset to guide future design toward physically viable materials, while quantifying the computational trade-offs of large-scale phonon screening. This work thus shifts the focus of crystal generation evaluation from thermodynamic stability to dynamical stability, with clear guidance for improving future generative methods.

Abstract

In this work, we introduce PhononBench, the first large-scale benchmark for dynamical stability in AI-generated crystals. Leveraging the recently developed MatterSim interatomic potential, which achieves DFT-level accuracy in phonon predictions across more than 10,000 materials, PhononBench enables efficient large-scale phonon calculations and dynamical-stability analysis for 108,843 crystal structures generated by six leading crystal generation models. PhononBench reveals a widespread limitation of current generative models in ensuring dynamical stability: the average dynamical-stability rate across all generated structures is only 25.83%, with the top-performing model, MatterGen, reaching just 41.0%. Further case studies show that in property-targeted generation-illustrated here by band-gap conditioning with MatterGen--the dynamical-stability rate remains as low as 23.5% even at the optimal band-gap condition of 0.5 eV. In space-group-controlled generation, higher-symmetry crystals exhibit better stability (e.g., cubic systems achieve rates up to 49.2%), yet the average stability across all controlled generations is still only 34.4%. An important additional outcome of this study is the identification of 28,119 crystal structures that are phonon-stable across the entire Brillouin zone, providing a substantial pool of reliable candidates for future materials exploration. By establishing the first large-scale dynamical-stability benchmark, this work systematically highlights the current limitations of crystal generation models and offers essential evaluation criteria and guidance for their future development toward the design and discovery of physically viable materials. All model-generated crystal structures, phonon calculation results, and the high-throughput evaluation workflows developed in PhononBench will be openly released at https://github.com/xqh19970407/PhononBench

Figures (6)

• Figure 1: Systematic Evaluation of Dynamical Stability in Crystal Generation Models. (a) Workflow of this study. Eight generative models (CrystaLLM, MatterGen, DiffCSP, InvDesFlow-AL, CrystalFlow, CrystalFormer, etc.) were used to generate a total of 221,000 novel structures. After removing duplicates and post-processing for CIF validity, full phonon-spectrum calculations were performed using MatterSim combined with Phonopy on 108,843 successfully relaxed crystals. In total, 28,119 structures were found to be dynamically stable, corresponding to an overall stability ratio of 25.83%. (b) Based on the systematic phonon-spectrum evaluation of over 10,000 materials by Miguel A. L. Marques et al., we employed MatterSim-v1—which attains DFT-level accuracy with an average error smaller than the difference between PBE and PBEsol functionals—as the unified potential for all subsequent phonon calculations, ensuring consistency in evaluation standards. (c) Key results and model performance comparison. All generated models face challenges in achieving dynamical stability. The top three models are MatterGen (41.0%), InvDesFlow-AL (38.4%), and CrystalFormer (34.4%), all of which benefited from pretraining on large, stable datasets such as Alex20. In contrast, models trained on smaller datasets such as MP20—e.g., CrystalFlow (16.7%)—show significantly lower performance. The LLM-based CrystaLLM (3.0%) performs the worst, highlighting its current disadvantage relative to architectures such as graph neural networks for this task.
• Figure 2: Dynamical Stability Analysis of Space-Group-Constrained Crystal Generation. (a) Distribution of the generated crystal structures across the seven crystal systems and their dynamical stability ratios (dark blue: dynamically stable; purple: containing imaginary modes). The cubic system exhibits the highest stability (49.2%), whereas the triclinic system shows the lowest (17%). (b) Distribution of the number of elemental components in the generated materials, where ternary and quaternary compounds account for more than half of the dataset, consistent with the trend observed in MP20. (c) Tetragonal structure: $\text{Na}_2\text{Li}_3\text{CdSb}$ (d) Trigonal structure: $\text{TmTh}(\text{GeRh})_2$ (e) Monoclinic structure: $\text{Ca}(\text{Sm}_2\text{Sn})_3$ (f)–(h) Phonon spectra corresponding to (c)–(e). $\text{Na}_2\text{Li}_3\text{CdSb}$ and $\text{TmTh}(\text{GeRh})_2$ exhibit pronounced imaginary phonon modes, indicating dynamical instability, whereas $\text{Ca}(\text{Sm}_2\text{Sn})_3$ shows no imaginary modes and is thus dynamically stable. These examples illustrate the ability of CrystalFormer to generate materials with complex chemistries, while also highlighting the remaining challenges in ensuring dynamical stability. They underscore the importance of incorporating explicit stability constraints or performing posterior stability screening within the generation pipeline.
• Figure 3: Dynamical Stability Analysis of Property-Constrained Crystal Generation. (a) Dynamical stability ratios of generated materials under different band-gap constraints. Among the 33,210 structures subjected to phonon analysis, the $E_g = 4.5$ eV condition yields the lowest stability (11.6%), whereas $E_g = 0.5$ eV yields the highest (23.5%). The stabilities for the other settings are 15.3% for $E_g = 1.5$ eV and 13.3% for both $E_g = 2.5$ eV and $E_g = 3.5$ eV. Overall, the stability rate under band-gap conditioning remains low (15.6%), implying that subsequent phonon validation with QE or VASP incurs considerable computational cost and poses challenges for large-scale applications. (b) Distribution of the number of elemental components in the generated materials. (c)–(e) Three representative crystal structures generated by MatterGen under the $E_g = 0.5$ eV constraint: ZnCu(BO$_2$)$_2$, Ba$_8$As$_3$I$_4$Br, and NdCuAsO. (f)–(h) Phonon spectra corresponding to panels (c)–(e). All three structures exhibit pronounced imaginary (negative) frequencies across multiple phonon branches, indicating strong dynamical instabilities in their current configurations.
• Figure 4: Elemental distribution heatmap of dynamically stable crystals. The heatmap illustrates the chemical element distribution of 28,119 newly discovered crystal structures with phonon (dynamical) stability generated by the crystal generation model. Each element is colored according to its occurrence frequency in the stable crystal set, with darker colors indicating higher frequencies. The analysis shows that oxygen (O) is the most prevalent element (10,272 occurrences), followed by lithium (Li, 5,792) and fluorine (F, 5,236), whereas noble gas elements appear only rarely. This distribution clearly indicates that current generative models tend to predict stable compounds containing chemically active elements, in good agreement with established chemical intuition.
• Figure 5: Convergence of the dynamical stability rate for various crystal generative models as a function of the number of tested samples. The figure demonstrates that stability estimates converge as sample size increases, with errors becoming negligible above 4,000 samples, ensuring robust and fair model ranking.