ComProScanner: A multi-agent based framework for composition-property structured data extraction from scientific literature

TL;DR

ComProScanner addresses the challenge of transforming scattered, unstructured literature into machine-readable composition–property data by implementing a configurable multi-agent framework that combines LLMs, Retrieval-Augmented Generation, and domain-specific tools. The four-phase workflow—metadata retrieval, article collection, information extraction, and evaluation/dataset creation—enables end users to build, validate, and visualize datasets that include complex compositions and associated properties such as the piezoelectric coefficient $d_{33}$. In a thorough evaluation over 100 articles across 10 LLMs, the DeepSeek-V3-0324 model achieved the highest overall accuracy of $0.82$ with composition accuracy of $0.90$, while several Qwen and Llama models also performed competitively; results reveal model-specific strengths and the value of agentic evaluation. The framework also delivers rich visualizations and a knowledge graph, and demonstrates that most extracted data are not present in existing databases, underscoring ComProScanner’s potential to accelerate materials discovery through large-scale, automated data curation.

Abstract

Since the advent of various pre-trained large language models, extracting structured knowledge from scientific text has experienced a revolutionary change compared with traditional machine learning or natural language processing techniques. Despite these advances, accessible automated tools that allow users to construct, validate, and visualise datasets from scientific literature extraction remain scarce. We therefore developed ComProScanner, an autonomous multi-agent platform that facilitates the extraction, validation, classification, and visualisation of machine-readable chemical compositions and properties, integrated with synthesis data from journal articles for comprehensive database creation. We evaluated our framework using 100 journal articles against 10 different LLMs, including both open-source and proprietary models, to extract highly complex compositions associated with ceramic piezoelectric materials and corresponding piezoelectric strain coefficients (d33), motivated by the lack of a large dataset for such materials. DeepSeek-V3-0324 outperformed all models with a significant overall accuracy of 0.82. This framework provides a simple, user-friendly, readily-usable package for extracting highly complex experimental data buried in the literature to build machine learning or deep learning datasets.

Figures (6)

• Figure 1: Overall workflow diagram of ComProScanner framework, separated in four distinct operational phases, distinguished with four different colour regions: (a) metadata retrieval (yellow), (b) article collection (purple), (c) information extraction (green), and (d) evaluation, post-processing, and dataset creation (brown).
• Figure 2: Comprehensive workflow diagram of the CrewAI-based extraction system in ComProScanner, comprising five specialised agents. The process begins with a property identifier agent ((a) RAG Crew) that leverages Retrieval-Augmented Generation (RAG) technology to filter relevant articles. The remaining four agents are strategically organised into two parallel functional subgroups: one dedicated to composition data extraction ((b) Composition Crew Set) and the other focused on synthesis information collection ((c) Synthesis Crew Set). Each subgroup implements a sequential two-agent architecture—the first agent extracts raw data while the second performs formatting and standardisation. The workflow integrates two essential tools: RAGTool for discriminating between mere property mentions and actual quantitative property values, and MaterialParserTool for accurate processing of complex chemical formulations.
• Figure 3: Normalised classification metrics (Precision, Recall, and F1-score) for model Llama-3.3-70B-Instruct (best performing model considering only normalised metrics) to showcase the performance of ComProScanner's performance capability for the considered models using: semantic evaluation through the PhysBERT model (red bars), and agentic evaluation through the Gemini-2.5-Pro reasoning model (blue bars).
• Figure 4: Confusion matrix from agentic evaluation, showcasing all 9 evaluation parameters, such as weight-based overall accuracy (average of composition accuracy and synthesis accuracy), weight-based composition accuracy and weight-based synthesis accuracy, classification metrics (Precision, Recall, and F1-score), and normalised classification metrics (normalised Precision, normalised Recall, and normalised F1-score), across 10 different LLMs used in this study.
• Figure 5: Distribution of various data types across the evaluated 100 articles: (a) piezoelectric material families, (b) synthesis precursors, and (c) characterisation techniques. Similarity thresholds of 0.8 were applied for families and precursors, whilst 0.78 was used for characterisation techniques to group semantically similar items.