LLM-Based Information Extraction to Support Scientific Literature Research and Publication Workflows

TL;DR

This work tackles the challenge of information overload in scholarly literature by applying LLM-based in-context learning to extract domain-specific semantic concepts (e.g., research questions, methods, findings) from scientific papers. The approach emphasizes rapid domain adaptation, evaluated on a BPM corpus with open-weight and commercial models, and validated through a demonstrator UI and user studies. Key contributions include a practical, open-source extraction pipeline, insights from a pilot study and community workshop, and a plan to integrate structured extractions with knowledge graphs to enhance reviewing, searching, and publishing workflows. The findings show strong semantic alignment for free-text fields and highlight the need for transparency and robust traceability, paving the way for embedding-based retrieval and KG-informed publishing templates (e.g., for ORKG) to improve discoverability and reuse in scientific research.

Abstract

The increasing volume of scholarly publications requires advanced tools for efficient knowledge discovery and management. This paper introduces ongoing work on a system using Large Language Models (LLMs) for the semantic extraction of key concepts from scientific documents. Our research, conducted within the German National Research Data Infrastructure for and with Computer Science (NFDIxCS) project, seeks to support FAIR (Findable, Accessible, Interoperable, and Reusable) principles in scientific publishing. We outline our explorative work, which uses in-context learning with various LLMs to extract concepts from papers, initially focusing on the Business Process Management (BPM) domain. A key advantage of this approach is its potential for rapid domain adaptation, often requiring few or even zero examples to define extraction targets for new scientific fields. We conducted technical evaluations to compare the performance of commercial and open-source LLMs and created an online demo application to collect feedback from an initial user-study. Additionally, we gathered insights from the computer science research community through user stories collected during a dedicated workshop, actively guiding the ongoing development of our future services. These services aim to support structured literature reviews, concept-based information retrieval, and integration of extracted knowledge into existing knowledge graphs.

Figures (1)

• Figure 1: LLM-based demo extraction pipeline.