Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Cross-Scale Perception Ability of Large Multimodal Models in Material Science

Yuting Zheng, Zijian Chen, Qi Jia

Abstract

Unraveling the hierarchical structure-property relationships is the central challenge of materials science, necessitating the interpretation of data across vast physical scales from micro to macro. Despite the rapid integration of Large Multimodal Models (LMMs) into scientific workflows, existing scientific benchmarks primarily focus on general chart interpretation or isolated common-sense reasoning, failing to capture reasoning ability across intricate physical dimensions. To address this, we introduce CSMBench, a dataset comprising 1,041 high-quality figures curated from premier journals up to September 2025. CSMBench categorizes data into four scientifically distinct regimes: atomic, micro, meso, and macro scales, strictly aligning with the focus and definitions in materials study. Through open-ended figure description and multiple-choice caption matching tasks, we evaluate state-of-the-art open-source and closed-source models. Our analysis identifies that performance varies significantly across physical scales due to the distinct visual characteristics, highlighting the limitations of current generalist models and identifying critical directions for achieving hierarchical and accurate understanding in materials research. The CSMBench is publicly released at: https://huggingface.co/datasets/lututu/CSMBench.

Benchmarking Cross-Scale Perception Ability of Large Multimodal Models in Material Science

Abstract

Unraveling the hierarchical structure-property relationships is the central challenge of materials science, necessitating the interpretation of data across vast physical scales from micro to macro. Despite the rapid integration of Large Multimodal Models (LMMs) into scientific workflows, existing scientific benchmarks primarily focus on general chart interpretation or isolated common-sense reasoning, failing to capture reasoning ability across intricate physical dimensions. To address this, we introduce CSMBench, a dataset comprising 1,041 high-quality figures curated from premier journals up to September 2025. CSMBench categorizes data into four scientifically distinct regimes: atomic, micro, meso, and macro scales, strictly aligning with the focus and definitions in materials study. Through open-ended figure description and multiple-choice caption matching tasks, we evaluate state-of-the-art open-source and closed-source models. Our analysis identifies that performance varies significantly across physical scales due to the distinct visual characteristics, highlighting the limitations of current generalist models and identifying critical directions for achieving hierarchical and accurate understanding in materials research. The CSMBench is publicly released at: https://huggingface.co/datasets/lututu/CSMBench.
Paper Structure (18 sections, 4 figures, 2 tables)

This paper contains 18 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. CSMBench
  4. Data Collection
  5. Source Data Collection
  6. Data Preprocessing
  7. Human Filtering
  8. Task Design
  9. Open-ended Task
  10. Multiple-choice Task
  11. Experiment
  12. Experimental Setup
  13. Benchmark Candidates
  14. Metrics
  15. Main Results
  16. ...and 3 more sections

Figures (4)

  • Figure 1: An overview of CSMBench, which consists of three primary phases: source data collection, data processing, and task design. The pipeline aggregates figures from authoritative materials journals across four physical scales, ranging from the atomic level to the macroscopic level, which strictly adhere to fundamental materials science definitions. The bottom-left schematic highlights the intrinsic characteristics across scales, serving as the theoretical basis for our scale categorization. Through a hybrid workflow combining automated description matching and expert manual filtering, raw data is refined into a high-quality structured dataset. Finally, the framework establishes two evaluation tasks to assess cross-scale perception: open-ended figure description and multiple-choice caption matching.
  • Figure 2: Pearson correlation coefficients between the atom, micro, meso, and macro scales using the STS metric and LLM score representatively.
  • Figure 3: Performance comparison on pure morphological images and charted hybrid multi-pattern images.
  • Figure 4: A comparative case study of GPT-5.1 and Qwen2.5-VL-7B across the dimensions of object recognition, feature extraction, and principle reasoning.