Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Expert-Grounded Automatic Prompt Engineering for Extracting Lattice Constants of High-Entropy Alloys from Scientific Publications using Large Language Models

Shunshun Liu, Talon R. Booth, Yangfeng Ji, Wesley Reinhart, Prasanna V. Balachandran

TL;DR

These results establish feedback-guided prompt optimization as a low-cost, transferable methodology for reliable scientific data extraction, providing a scalable pathway for complex LLM-assisted research tasks.

Abstract

Large language models (LLMs) have shown promise for scientific data extraction from publications, but rely on manual prompt refinement. We present an expert-grounded automatic prompt optimization framework that enhances LLM entity extraction reliability. Using high-entropy alloy lattice constant extraction as a testbed, we optimized prompts for Claude 3.5 Sonnet through feedback cycles on seven expert-annotated publications. Despite a modest optimization budget, recall improved from 0.27 to > 0.9, demonstrating that a small, expert-curated dataset can yield significant improvements. The approach was applied to extract lattice constants from 2,267 publications, yielding data for 1,861 compositions. The optimized prompt transferred effectively to newer models: Claude 4.5 Sonnet, GPT-5, and Gemini 2.5 Flash. Analysis revealed three categories of LLM mistakes: contextual hallucination, semantic misinterpretation, and unit conversion errors, emphasizing the need for validation protocols. These results establish feedback-guided prompt optimization as a low-cost, transferable methodology for reliable scientific data extraction, providing a scalable pathway for complex LLM-assisted research tasks.

Expert-Grounded Automatic Prompt Engineering for Extracting Lattice Constants of High-Entropy Alloys from Scientific Publications using Large Language Models

TL;DR

These results establish feedback-guided prompt optimization as a low-cost, transferable methodology for reliable scientific data extraction, providing a scalable pathway for complex LLM-assisted research tasks.

Abstract

Large language models (LLMs) have shown promise for scientific data extraction from publications, but rely on manual prompt refinement. We present an expert-grounded automatic prompt optimization framework that enhances LLM entity extraction reliability. Using high-entropy alloy lattice constant extraction as a testbed, we optimized prompts for Claude 3.5 Sonnet through feedback cycles on seven expert-annotated publications. Despite a modest optimization budget, recall improved from 0.27 to > 0.9, demonstrating that a small, expert-curated dataset can yield significant improvements. The approach was applied to extract lattice constants from 2,267 publications, yielding data for 1,861 compositions. The optimized prompt transferred effectively to newer models: Claude 4.5 Sonnet, GPT-5, and Gemini 2.5 Flash. Analysis revealed three categories of LLM mistakes: contextual hallucination, semantic misinterpretation, and unit conversion errors, emphasizing the need for validation protocols. These results establish feedback-guided prompt optimization as a low-cost, transferable methodology for reliable scientific data extraction, providing a scalable pathway for complex LLM-assisted research tasks.
Paper Structure (17 sections, 2 equations, 4 figures, 3 tables)

This paper contains 17 sections, 2 equations, 4 figures, 3 tables.

Figures (4)

  • Figure 1: Schematic workflow for expert-grounded prompt optimization and large-scale data extraction. Numbers in parentheses indicate the sequential workflow steps, with corresponding arrow colors. The details of each module are described in the main text.
  • Figure 2: Full initial prompt. A machine-readable version is provided in the Figure S2 in SI.
  • Figure 3: A condensed and streamlined visualization of the complete optimized prompt, broken down into four panels: (a) Task instructions, the scope of extraction, and the definition of the data extraction scheme. (b) An example of the extracted data in JSON format. (c) Additional instructions along with a JSON example demonstrating the correct output. (d) A JSON example illustrating how to handle missing values. A machine-readable version is provided in the Figure S3 in SI.
  • Figure 4: Overview of extracted HEA data from 2,267 publications. (a) A pie-chart showing the relative count of entries with and without lattice constant data: of 4,648 total extracted compositions, 1,861 (40.0%) contained lattice constant while 2,787 (60.0%) lacked this data. (b) A pie-chart showing the phase distribution among the 1,865 entries with lattice constants: 311 single-phase BCC (16.7%), 408 single-phase FCC (21.9%), and 1,142 other phases (61.4%), including multi-phase mixtures, ordered structures, and amorphous structures. (c) A pie-chart showing the processing condition distribution for the 311 single-phase BCC alloys: 186 as-cast (59.8%), 111 processed by other methods, such as selective laser melting and powder processing, or included post-processing heat treatments, such as annealing (35.7%), and 14 with unreported processing information (4.5%). (d) Histogram showing lattice constant distribution for the 186 as-cast single-phase BCC alloys. Most values fall within the range of 2.8–3.5 Å for BCC structures. There were 21 entries near 0.3 Å that represents a systematic unit conversion error where the LLM failed to convert nanometer values to Å unit.