MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science

TL;DR

MatSciBench presents a comprehensive benchmark of 1340 materials-science questions across 6 fields and 31 sub-fields, with 3-level difficulty and 315 multimodal items, enabling fine-grained evaluation of LLM reasoning. The study benchmarks six thinking models and five non-thinking models, showing no single approach dominates and that tool augmentation often helps while self-correction yields mixed results. Through multi-dimensional analyses of difficulty, efficiency, multimodal performance, and failure patterns, the work reveals both capabilities and limitations of current LLMs in materials science reasoning and highlights the nuanced role of retrieval-augmented generation. Overall, MatSciBench establishes a rigorous platform to drive improvements in domain-specific scientific reasoning and multimodal understanding for materials science applications.

Abstract

Large Language Models (LLMs) have demonstrated remarkable abilities in scientific reasoning, yet their reasoning capabilities in materials science remain underexplored. To fill this gap, we introduce MatSciBench, a comprehensive college-level benchmark comprising 1,340 problems that span the essential subdisciplines of materials science. MatSciBench features a structured and fine-grained taxonomy that categorizes materials science questions into 6 primary fields and 31 sub-fields, and includes a three-tier difficulty classification based on the reasoning length required to solve each question. MatSciBench provides detailed reference solutions enabling precise error analysis and incorporates multimodal reasoning through visual contexts in numerous questions. Evaluations of leading models reveal that even the highest-performing model, Gemini-2.5-Pro, achieves under 80% accuracy on college-level materials science questions, highlighting the complexity of MatSciBench. Our systematic analysis of different reasoning strategie--basic chain-of-thought, tool augmentation, and self-correction--demonstrates that no single method consistently excels across all scenarios. We further analyze performance by difficulty level, examine trade-offs between efficiency and accuracy, highlight the challenges inherent in multimodal reasoning tasks, analyze failure modes across LLMs and reasoning methods, and evaluate the influence of retrieval-augmented generation. MatSciBench thus establishes a comprehensive and solid benchmark for assessing and driving improvements in the scientific reasoning capabilities of LLMs within the materials science domain.

Figures (12)

• Figure 1: Taxonomy of MatSciBench Materials Science QAs.
• Figure 2: Difficulty Distribution by Taxonomy Primary Fields.
• Figure 3: The Performance of LLMs across Difficulty Levels.
• Figure 4: The Average Output Length v.s. The Accuracies.
• Figure 5: The Performance Comparison of MLLM between Questions w and w/o Images.