Towards an automated workflow in materials science for combining multi-modal simulative and experimental information using data mining and large language models

TL;DR

The paper addresses the challenge that valuable information in materials science literature is embedded in PDFs and multi-modal formats, hindering machine-readable access. It proposes an automated workflow that combines OCR-driven extraction, incorporation of user-specific microstructure data, and transformer-based models to build a multi-modal database, which feeds a Retrieval-Augmented Generation (RAG) based Large Language Model (LLM) for fast, context-aware Q&A (MINDQUEST). Key contributions include end-to-end data extraction and structuring from text, equations, images, and metadata; storage in a vector database with embedding-based retrieval; and the deployment of a RAG-based chat bot for literature plus local data querying in dislocation-microstructure research within FCC single crystals. This approach accelerates information retrieval and knowledge synthesis in materials science and lays groundwork for ontology integration and direct multi-modal model deployment in future work.

Abstract

To retrieve and compare scientific data of simulations and experiments in materials science, data needs to be easily accessible and machine readable to qualify and quantify various materials science phenomena. The recent progress in open science leverages the accessibility to data. However, a majority of information is encoded within scientific documents limiting the capability of finding suitable literature as well as material properties. This manuscript showcases an automated workflow, which unravels the encoded information from scientific literature to a machine readable data structure of texts, figures, tables, equations and meta-data, using natural language processing and language as well as vision transformer models to generate a machine-readable database. The machine-readable database can be enriched with local data, as e.g. unpublished or private material data, leading to knowledge synthesis. The study shows that such an automated workflow accelerates information retrieval, proximate context detection and material property extraction from multi-modal input data exemplarily shown for the research field of microstructural analyses of face-centered cubic single crystals. Ultimately, a Retrieval-Augmented Generation (RAG) based Large Language Model (LLM) enables a fast and efficient question answering chat bot.

Figures (13)

• Figure 1: Number of publications in material science during the past 50 years for different queries based on data from dimension.ai appdimensions.
• Figure 1: An example of the layout detection tool surya from the marker OCR model structuring the data into text, sections, tables, figures and equations (here, Xu_2016 reproduced with permission from IOP Publishing under a Creative Commons License).
• Figure 2: Automated workflow to generate a Retrieval-Augmented Generation (RAG) based Large Language Model (LLM) using a multi-modal database.
• Figure 3: Example images of a DDD and a CDD microstructure (given as 2d slice of a 3d material system) including metadata information about the simulation features.
• Figure 4: Example for deciphering full-text PDF document into structured database entities. Each document is subdivided into the following entities: markdown text, meta-data, equations, tables and images. (here, Xu_2016 reproduced with permission from IOP Publishing under a Creative Commons License)