MATRIX: A Multimodal Benchmark and Post-Training Framework for Materials Science

TL;DR

MATRIX introduces a multimodal benchmark and post-training framework for materials science reasoning that combines theory, coursework-style prompts, and real experimental imagery (SEM, XRD, EDS, TGA). It demonstrates that aligned visual grounding during post-training reshapes shared representations, yielding notable gains in both image-based interpretation (10–25%) and text-only scientific reasoning (5–16%), with alignment disruptions destroying these benefits. The approach shows transfer of multimodal training benefits to other scientific domains (ScienceQA, PubMedQA, SciBench), though gains are selective toward applied, explanation-driven tasks rather than formal derivations. By releasing a controlled, rubric-based evaluation alongside a disjoint post-training dataset (MATRIX-PT), the work provides a diagnostic toolkit to study cross-modal reasoning and the role of visual grounding in scientific AI systems.

Abstract

Scientific reasoning in materials science requires integrating multimodal experimental evidence with underlying physical theory. Existing benchmarks make it difficult to assess whether incorporating visual experimental data during post-training improves mechanism-grounded explanation reasoning beyond text-only supervision. We introduce MATRIX, a multimodal benchmark for materials science reasoning that evaluates foundational theory, research-level reasoning, and the interpretation of real experimental artifacts across multiple characterization modalities. Using MATRIX as a controlled diagnostic, we isolate the effect of visual grounding by comparing post-training on structured materials science text alone with post-training that incorporates paired experimental images. Despite using relatively small amounts of multimodal data, visual supervision improves experimental interpretation by 10-25% and yields 5-16% gains on text-only scientific reasoning tasks. Our results demonstrate that these improvements rely on correct image-text alignment during post-training, highlighting cross-modal representational transfer. We also observe consistent improvements on ScienceQA and PubMedQA, demonstrating that the benefits of structured multimodal post-training extend beyond materials science. The MATRIX dataset is available at https://huggingface.co/datasets/radical-ai/MATRIX and the model at https://huggingface.co/radical-ai/MATRIX-PT.

Figures (9)

• Figure 1: Breakdown of MATRIX benchmark by text and image tasks, further broken down by subtasks.
• Figure 2: MATRIX data curation and task construction pipeline. Research papers are parsed into page-level text and experimental figure streams. Limited early-paper context is used to construct hypothesis-generation prompts, while experimental figures (SEM-BSE, SEM-SE, XRD, EDS, TGA) are combined with their local textual context to construct image–caption pairs. These pairs are standardized to support standalone experimental reasoning without reliance on full-paper memorization. Validation and quality-control steps ensure that captions remain grounded in observable evidence. See \ref{['sec:appendix_data']} for full implementation details and filtering criteria.
• Figure 3: Each row illustrates a controlled modification of the question–image–answer structure used during post-training. Top: The answer text is permuted while the question and image remain fixed. Middle: The input image is permuted while the question and answer text remain fixed. Bottom: The image is removed entirely. These controls test whether observed gains arise from correct image–text alignment rather than spurious correlations or answer memorization.
• Figure 4: Example SEM-BSE image–caption pair and enriched description generated from the source corpus reck2019fatigue.
• Figure 5: Example SEM-SE image–caption pair and enriched description generated from the source corpus spencer2017comparing.