Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QMBench: A Research Level Benchmark for Quantum Materials Research

Yanzhen Wang, Yiyang Jiang, Diana Golovanova, Kamal Das, Hyeonhu Bae, Yufei Zhao, Huu-Thong Le, Abhinava Chatterjee, Yunzhe Liu, Chao-Xing Liu, Felipe H. da Jornada, Binghai Yan, Xiao-Liang Qi

TL;DR

QMBench introduces a research-level benchmark for evaluating AI agents in quantum materials science by spanning five problem domains—structural, symmetry, computational methodologies, electronic, and thermal/optical/magnetic properties—and by requiring practical outputs such as POSCAR files alongside traditional answer formats. The benchmark emphasizes DFT-centered workflows and provides a community platform for ongoing updates and collaborative development. Empirical results show a clear split: current models excel at recall-based, knowledge-oriented tasks but struggle with rigorous derivations, quantitative figure interpretation, and complex atomistic manipulations, with multimodal models outperforming text-only ones yet still far from robust research-level performance. These findings highlight concrete gaps to close for AI scientists in quantum materials, and they position bench.science as a vehicle to iteratively improve evaluation and tooling for future AI-driven materials research.

Abstract

We introduce QMBench, a comprehensive benchmark designed to evaluate the capability of large language model agents in quantum materials research. This specialized benchmark assesses the model's ability to apply condensed matter physics knowledge and computational techniques such as density functional theory to solve research problems in quantum materials science. QMBench encompasses different domains of the quantum material research, including structural properties, electronic properties, thermodynamic and other properties, symmetry principle and computational methodologies. By providing a standardized evaluation framework, QMBench aims to accelerate the development of an AI scientist capable of making creative contributions to quantum materials research. We expect QMBench to be developed and constantly improved by the research community.

QMBench: A Research Level Benchmark for Quantum Materials Research

TL;DR

QMBench introduces a research-level benchmark for evaluating AI agents in quantum materials science by spanning five problem domains—structural, symmetry, computational methodologies, electronic, and thermal/optical/magnetic properties—and by requiring practical outputs such as POSCAR files alongside traditional answer formats. The benchmark emphasizes DFT-centered workflows and provides a community platform for ongoing updates and collaborative development. Empirical results show a clear split: current models excel at recall-based, knowledge-oriented tasks but struggle with rigorous derivations, quantitative figure interpretation, and complex atomistic manipulations, with multimodal models outperforming text-only ones yet still far from robust research-level performance. These findings highlight concrete gaps to close for AI scientists in quantum materials, and they position bench.science as a vehicle to iteratively improve evaluation and tooling for future AI-driven materials research.

Abstract

We introduce QMBench, a comprehensive benchmark designed to evaluate the capability of large language model agents in quantum materials research. This specialized benchmark assesses the model's ability to apply condensed matter physics knowledge and computational techniques such as density functional theory to solve research problems in quantum materials science. QMBench encompasses different domains of the quantum material research, including structural properties, electronic properties, thermodynamic and other properties, symmetry principle and computational methodologies. By providing a standardized evaluation framework, QMBench aims to accelerate the development of an AI scientist capable of making creative contributions to quantum materials research. We expect QMBench to be developed and constantly improved by the research community.
Paper Structure (14 sections, 1 equation, 1 figure, 2 tables)

This paper contains 14 sections, 1 equation, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Detailed description of QMBench
  3. Problem Statistics
  4. Structural Properties
  5. Symmetry Principles
  6. Computational Methodologies
  7. Electronic Properties
  8. Thermal, Optical and Magnetic Properties
  9. Evaluation Method
  10. Model Performance
  11. Statistics
  12. Further analysis to model performance
  13. Related Works
  14. Conclusion and Discussion

Figures (1)

  • Figure 1: Performance of multimodal LLMs on QMBench for questions at different difficulty levels. (a) Performance on questions at different difficulty levels. (b) Performance on questions of different categories.