AutoMat: Enabling Automated Crystal Structure Reconstruction from Microscopy via Agentic Tool Use

TL;DR

AutoMat addresses the bottleneck of turning atomic-resolution STEM images into simulation-ready crystal structures and property predictions by fusing pattern-adaptive denoising, physics-guided template retrieval, symmetry-constrained reconstruction, and fast ML-based relaxation within an LLM-driven agent. The approach outputs CIF files and property estimates in an end-to-end fashion, validated on the STEM2Mat-Bench with 450 representative image–structure pairs. Key contributions include the MOE-DIVAESR denoiser, STEM2CIF reconstruction, MatterSim-based property prediction, and a benchmark tailored to STEM-to-material pipelines. AutoMat substantially outperforms existing multimodal models on lattice and energy metrics, demonstrating the feasibility of bridging microscopy and atomistic simulation for scalable materials discovery, with code and data openly released.

Abstract

Machine learning-based interatomic potentials and force fields depend critically on accurate atomic structures, yet such data are scarce due to the limited availability of experimentally resolved crystals. Although atomic-resolution electron microscopy offers a potential source of structural data, converting these images into simulation-ready formats remains labor-intensive and error-prone, creating a bottleneck for model training and validation. We introduce AutoMat, an end-to-end, agent-assisted pipeline that automatically transforms scanning transmission electron microscopy (STEM) images into atomic crystal structures and predicts their physical properties. AutoMat combines pattern-adaptive denoising, physics-guided template retrieval, symmetry-aware atomic reconstruction, fast relaxation and property prediction via MatterSim, and coordinated orchestration across all stages. We propose the first dedicated STEM2Mat-Bench for this task and evaluate performance using lattice RMSD, formation energy MAE, and structure-matching success rate. By orchestrating external tool calls, AutoMat enables a text-only LLM to outperform vision-language models in this domain, achieving closed-loop reasoning throughout the pipeline. In large-scale experiments over 450 structure samples, AutoMat substantially outperforms existing multimodal large language models and tools. These results validate both AutoMat and STEM2Mat-Bench, marking a key step toward bridging microscopy and atomistic simulation in materials science.The code and dataset are publicly available at https://github.com/yyt-2378/AutoMat and https://huggingface.co/datasets/yaotianvector/STEM2Mat.

Figures (4)

• Figure 1: Overview of AutoMat. The left part shows an LLM agent how to manage four modules—pattern-adaptive denoising, template selection, atomic reconstruction (STEM2CIF), and ML-based property prediction. The right panel presents a line chart and radar plot comparing different models in terms of energy and structural errors.
• Figure 2: Overview of the STEM2Mat Benchmark design, illustrating the tiered classification of STEM samples by material complexity and imaging dose, which systematically stratifies reconstruction difficulty from simple unary to complex ternary compounds.
• Figure 3: AutoMat’s LLM agent orchestrates four stages—denoising, template matching, structure reconstruction, and property prediction—from STEM image to relaxed crystal with properties.
• Figure 4: Case studies on Tier 2 and Tier 3 samples comparing AutoMat, GPT-4.1mini and Llama-4-Maverick: enhanced images, best‐matched templates, reconstructed unit cells and predicted energies.