GSAP-ERE: Fine-Grained Scholarly Entity and Relation Extraction Focused on Machine Learning

TL;DR

GSAP-ERE addresses the need for fine-grained scholarly information extraction in machine learning by presenting a manually annotated full-text dataset with 10 entity types and 18 relation types, spanning 63K entities and 35K relations across 100 publications. The authors compare supervised PLM-based baselines (pipeline and joint) against unsupervised LLM prompting, showing that fine-tuned models substantially outperform prompting approaches in both NER and RE tasks. They also explore a hypergraph-based joint extraction method (HGERE) and demonstrate that it yields strong results, outperforming the pipeline baseline. The work further provides a detailed annotation scheme, quality control procedures, and a public release plan, highlighting GSAP-ERE as a key resource for knowledge-graph construction, data provenance tracing, and scalable reproducibility assessment in ML research.

Abstract

Research in Machine Learning (ML) and AI evolves rapidly. Information Extraction (IE) from scientific publications enables to identify information about research concepts and resources on a large scale and therefore is a pathway to improve understanding and reproducibility of ML-related research. To extract and connect fine-grained information in ML-related research, e.g. method training and data usage, we introduce GSAP-ERE. It is a manually curated fine-grained dataset with 10 entity types and 18 semantically categorized relation types, containing mentions of 63K entities and 35K relations from the full text of 100 ML publications. We show that our dataset enables fine-tuned models to automatically extract information relevant for downstream tasks ranging from knowledge graph (KG) construction, to monitoring the computational reproducibility of AI research at scale. Additionally, we use our dataset as a test suite to explore prompting strategies for IE using Large Language Models (LLM). We observe that the performance of state-of-the-art LLM prompting methods is largely outperformed by our best fine-tuned baseline model (NER: 80.6%, RE: 54.0% for the fine-tuned model vs. NER: 44.4%, RE: 10.1% for the LLM). This disparity of performance between supervised models and unsupervised usage of LLMs suggests datasets like GSAP-ERE are needed to advance research in the domain of scholarly information extraction.