SCALAR: Quantifying Structural Hallucination, Consistency, and Reasoning Gaps in Materials Foundation Models

TL;DR

SCALAR tackles how structural hallucination and reasoning gaps emerge when materials representations are scaled from bulk unit cells to finite nanoparticles. It introduces a cross-scale benchmark with three tasks—CIF→property prediction, physics-grounded Chain-of-Thought reasoning, and inverse retrieval—and a comprehensive set of metrics to capture hallucination, cross-scale consistency, and physically grounded reasoning under radii $R\in\{10,\dots,30\}$ Å. The dataset is constructed from a $20\times20\times20$ supercell with spherical carving, SO(3) rotation sampling, and split-aware ID/OOD regimes, enabling controlled analysis of cross-scale invariants and geometry-driven failures. Across multiple foundation models, results show that explicit reasoning can reduce some errors but often destabilizes consistency and validity, demonstrating that geometric scale generalization cannot be inferred from accuracy alone and that principled, scale-aware evaluation is critical for robust materials reasoning.

Abstract

Large language models are increasingly applied to materials science reasoning, yet their behavior under physically structured distribution shifts remains poorly understood. We introduce SCALAR (Structural Consistency And Logic Across Regimes), a benchmark for evaluating geometric scale generalization and its connection to structural hallucination, consistency, and reasoning in materials foundation models. Given canonical crystal representations, models must reason about derived nanoparticle structures obtained through supercell expansion and geometric truncation across length scales spanning a few atoms to over 18,000 atoms, totaling $\approx$100,000 structures from DFT-validated unit cells. SCALAR defines three tasks. (i) CIF to property prediction. (ii) A Chain-of-Thought variant with explicit physics-grounded reasoning. (iii) Inverse retrieval identifying crystals from candidates given target properties. Outputs are evaluated via structured metrics capturing numeric error, hallucination, cross-prompt consistency, monotonic reasoning, output validity, and retrieval regret. Experiments across diverse foundation models reveal large, model-dependent shifts under explicit reasoning, often reducing hallucination and error, but frequently destabilizing consistency or validity. These results demonstrate that geometric scale generalization cannot be inferred from accuracy alone. Supplementary materials are available at https://github.com/KurbanIntelligenceLab/SCALAR.

Figures (5)

• Figure 1: Systematic benchmark construction pipeline. Phase I constructs finite nanoparticles $\mathcal{C}_R$ from DFT-relaxed unit cells via supercell replication and spherical carving at radii $R\in\{10,\ldots,30\}$ Å. Phase II samples rigid rotations on $\mathrm{SO}(3)$ using unit quaternions with minimum geodesic spacing $\vartheta$, then applies split-specific offsets. Phase III enforces split-aware exclusion: ID rotations maintain margin $\varepsilon_{\mathrm{ID}}$ from training, OOD rotations maintain margin $\varepsilon_{\mathrm{OD}}$ from both training and ID sets, with radius-based partitioning ($\mathcal{S}_{\mathrm{ID}}$, $\mathcal{S}_{\mathrm{OD}}$).
• Figure 2: Columns (left to right) correspond to Ag, RbBH$_{3}$, CH$_{3}$NH$_{3}$PbI$_{3}$, TiO$_{2}$, Fe$_{2}$O$_{3}$. Rows show the reference unit cell (top), a nanoparticle carved at $R=10$ Å (middle), and a nanoparticle carved at $R=30$ Å (bottom). Each panel reports the total atom count ($N$) and elemental composition. Across the dataset, particle sizes span more than four orders of magnitude, from a minimum of $N=4$ atoms for the smallest particles to a maximum of $N=18,123$ atoms for the largest configurations. For the element columns, the value above each element is its percent of total atoms in the dataset, and the value below each element is the average atoms per structure containing that element. Atom colors follow the standard CPK scheme.
• Figure 3: Distributions and pairwise relationships of lattice parameters ($a$, $b$, $c$) in the SCALAR dataset. Diagonal panels show kernel density estimates annotated with summary statistics, highlighting substantial variability and anisotropy across unit cells, with $b$ exhibiting the largest dispersion. Off-diagonal panels show pairwise scatter plots with Pearson correlation coefficients.
• Figure 4: Material-level sensitivity analysis for geometric scale extrapolation. Rows correspond to 0-, 1-, 3-shot, and columns to mean nearest-neighbor distance, unit cell volume, density. For each regime, we show the top-3 best-performing (green) and worst-3 performing (red) materials. Circles (ID) and squares (OOD) are connected by lines whose length and direction indicate the extrapolation gap magnitude and sign.
• Figure 5: CoT prompting effect analysis. Horizontal diverging bar charts show the CoT impact (log-scaled $\Delta$) for ten models across three metrics under 1-shot and 3-shot regimes. Bars extend from a central zero line: leftward bars (negative $\Delta$, green) indicate that CoT improves the target metric, while rightward bars (positive $\Delta$, red) indicate that CoT worsens it. Metric direction differs by outcome: for Errors and Consistency, decreases are beneficial ($\downarrow$ better), whereas for Reasoning, increases are beneficial ($\uparrow$ better).