Extracting Polymer Nanocomposite Samples from Full-Length Documents

TL;DR

This work tackles the problem of automatically extracting $N$-ary polymer nanocomposite samples from full-length materials science articles. It introduces PNCExtract, a benchmark with a dual evaluation framework (Partial-F1 and Strict-F1) and a NanoMine-derived dataset, and systematically compares End-to-End and NER+RE prompting strategies, augmented by self-consistency and dense retrieval for long documents. The results show that GPT-4 Turbo in a zero-shot setting can surpass some baselines but still misses a substantial portion of samples, with key error modes including composition complexity and non-standard chemical names. Overall, the paper advances automated, scalable extraction of materials data from full texts and points to multimodal and enhanced normalization techniques as promising directions for future work.

Abstract

This paper investigates the use of large language models (LLMs) for extracting sample lists of polymer nanocomposites (PNCs) from full-length materials science research papers. The challenge lies in the complex nature of PNC samples, which have numerous attributes scattered throughout the text. The complexity of annotating detailed information on PNCs limits the availability of data, making conventional document-level relation extraction techniques impractical due to the challenge in creating comprehensive named entity span annotations. To address this, we introduce a new benchmark and an evaluation technique for this task and explore different prompting strategies in a zero-shot manner. We also incorporate self-consistency to improve the performance. Our findings show that even advanced LLMs struggle to extract all of the samples from an article. Finally, we analyze the errors encountered in this process, categorizing them into three main challenges, and discuss potential strategies for future research to overcome them.

Figures (5)

• Figure 1: A snippet from a PNC research article DALMAS2007829 and the extracted PNC sample list from the NanoMine database. Note how information for a single sample is extracted from multiple parts of the article text.
• Figure 2: Two prompting strategies for PNC sample extraction with LLM are presented. On the left, the end-to-end (E2E) approach uses a single prompt to directly extract PNC samples. On the right, the NER+RE approach first identifies relevant entities and then classifies their relations through yes/no prompts to validate PNC samples.
• Figure 3: Comparison of Micro Partial F1 Scores Across Different Models and Document Lengths. "Top 5", "Top 10", and "Top 30" indicate document summaries retrieved with $k$ set to $5$, $10$, and $30$ respectively.
• Figure 4: Examples of challenges for LLMs, showcasing three categories of challenges encountered in capturing accurate PNC sample compositions. Each row demonstrates a specific challenge, the ground-truth sample, the model's prediction, and a brief explanation of the issue."
• Figure 5: An inconsistent sample in NanoMine that we exclude from our dataset.