Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering

Maciej P. Polak, Dane Morgan

TL;DR

The paper tackles the bottleneck of manual data extraction from research papers by introducing ChatExtract, a zero-shot workflow that leverages engineered prompts within conversational LLMs to reliably extract Material-Value-Unit triplets. By combining relevance classification, carefully structured multi-step questioning, and redundancy through follow-up prompts, the method significantly mitigates hallucinations and improves data fidelity, achieving high precision and recall across tested properties (e.g., up to $P\approx90.8\%$, $R\approx87.7\%$ for bulk modulus with GPT-4). The approach is demonstrated on multiple datasets, including critical cooling rates for metallic glasses and yield strengths for high-entropy alloys, producing raw, cleaned, and standardized databases and showing strong transferability across models and properties. The results suggest that ChatExtract can simplify and accelerate large-scale materials databases development, with potential for broader adoption as LLMs evolve. Overall, the study provides a practical, transferable framework for automated, high-quality data extraction from scientific literature.

Abstract

There has been a growing effort to replace manual extraction of data from research papers with automated data extraction based on natural language processing, language models, and recently, large language models (LLMs). Although these methods enable efficient extraction of data from large sets of research papers, they require a significant amount of up-front effort, expertise, and coding. In this work we propose the ChatExtract method that can fully automate very accurate data extraction with minimal initial effort and background, using an advanced conversational LLM. ChatExtract consists of a set of engineered prompts applied to a conversational LLM that both identify sentences with data, extract that data, and assure the data's correctness through a series of follow-up questions. These follow-up questions largely overcome known issues with LLMs providing factually inaccurate responses. ChatExtract can be applied with any conversational LLMs and yields very high quality data extraction. In tests on materials data we find precision and recall both close to 90% from the best conversational LLMs, like ChatGPT-4. We demonstrate that the exceptional performance is enabled by the information retention in a conversational model combined with purposeful redundancy and introducing uncertainty through follow-up prompts. These results suggest that approaches similar to ChatExtract, due to their simplicity, transferability, and accuracy are likely to become powerful tools for data extraction in the near future. Finally, databases for critical cooling rates of metallic glasses and yield strengths of high entropy alloys are developed using ChatExtract.

Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering

TL;DR

The paper tackles the bottleneck of manual data extraction from research papers by introducing ChatExtract, a zero-shot workflow that leverages engineered prompts within conversational LLMs to reliably extract Material-Value-Unit triplets. By combining relevance classification, carefully structured multi-step questioning, and redundancy through follow-up prompts, the method significantly mitigates hallucinations and improves data fidelity, achieving high precision and recall across tested properties (e.g., up to , for bulk modulus with GPT-4). The approach is demonstrated on multiple datasets, including critical cooling rates for metallic glasses and yield strengths for high-entropy alloys, producing raw, cleaned, and standardized databases and showing strong transferability across models and properties. The results suggest that ChatExtract can simplify and accelerate large-scale materials databases development, with potential for broader adoption as LLMs evolve. Overall, the study provides a practical, transferable framework for automated, high-quality data extraction from scientific literature.

Abstract

There has been a growing effort to replace manual extraction of data from research papers with automated data extraction based on natural language processing, language models, and recently, large language models (LLMs). Although these methods enable efficient extraction of data from large sets of research papers, they require a significant amount of up-front effort, expertise, and coding. In this work we propose the ChatExtract method that can fully automate very accurate data extraction with minimal initial effort and background, using an advanced conversational LLM. ChatExtract consists of a set of engineered prompts applied to a conversational LLM that both identify sentences with data, extract that data, and assure the data's correctness through a series of follow-up questions. These follow-up questions largely overcome known issues with LLMs providing factually inaccurate responses. ChatExtract can be applied with any conversational LLMs and yields very high quality data extraction. In tests on materials data we find precision and recall both close to 90% from the best conversational LLMs, like ChatGPT-4. We demonstrate that the exceptional performance is enabled by the information retention in a conversational model combined with purposeful redundancy and introducing uncertainty through follow-up prompts. These results suggest that approaches similar to ChatExtract, due to their simplicity, transferability, and accuracy are likely to become powerful tools for data extraction in the near future. Finally, databases for critical cooling rates of metallic glasses and yield strengths of high entropy alloys are developed using ChatExtract.
Paper Structure (8 sections, 1 equation, 3 figures, 1 table)

This paper contains 8 sections, 1 equation, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Results and Discussion
  3. Description of the Data Extraction Workflow
  4. Performance Evaluation and Model Comparison
  5. Application to Tables and Figures
  6. Results of Real-life Data Extraction
  7. Conclusions
  8. Methods

Figures (3)

  • Figure 1: A simplified flowchart describing our ChatExtract method of extracting structured data using a conversational large language model. Only the key ideas for each of the steps are shown, with the fully detailed workflow presented in Fig. \ref{['fig:flowchart']}
  • Figure 2: A flowchart describing our ChatExtract method of extracting structured data using a conversational large language model. Blue boxes represent prompts given to the model, grey boxes are instructions to the user, "Yes", "No", and "None" boxes are model's responses. The bold text in "[]" are to be replaced with appropriate values of the named item, which includes one of sentence (the target sentence being analyzed); text (the expanded text consisting of Title, Preceding sentence, and target sentence); name of the property; extracted material, value, or unit.
  • Figure 3: A flowchart describing our ChatExtract method of extracting structured data from: (a) tables, (b) figures; using a conversational large language model. Blue boxes represent prompts given to the model, grey boxes are instructions to the user, "Yes", "No" boxes are model's responses. The bold text in "[]" are to be replaced with appropriate values of the named item.