SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents

TL;DR

A new entity and relation extraction dataset for entities related to datasets, methods, and tasks in scientific articles, containing 106 manually annotated full-text scientific publications with over 24k entities and 12k relations is released.

Abstract

Scientific information extraction (SciIE) is critical for converting unstructured knowledge from scholarly articles into structured data (entities and relations). Several datasets have been proposed for training and validating SciIE models. However, due to the high complexity and cost of annotating scientific texts, those datasets restrict their annotations to specific parts of paper, such as abstracts, resulting in the loss of diverse entity mentions and relations in context. In this paper, we release a new entity and relation extraction dataset for entities related to datasets, methods, and tasks in scientific articles. Our dataset contains 106 manually annotated full-text scientific publications with over 24k entities and 12k relations. To capture the intricate use and interactions among entities in full texts, our dataset contains a fine-grained tag set for relations. Additionally, we provide an out-of-distribution test set to offer a more realistic evaluation. We conduct comprehensive experiments, including state-of-the-art supervised models and our proposed LLM-based baselines, and highlight the challenges presented by our dataset, encouraging the development of innovative models to further the field of SciIE.

Figures (5)

• Figure 1: Top: An annotation sample of our SciER dataset, illustrating the labeling process and data structure. The sentence $S_1$ contains two annotated spans denoting two entities $E_1$ and $E_2$, with respective types METHOD and TASK. Bottom: A table detailing the input and output of the three tasks supported by our SciER dataset, including Named Entity Recognition (NER), Relation Extraction (RE), and Entity and Relation Extraction (ERE).
• Figure 2: Overall architecture of LLM in-context learning (few-shot) baselines for NER, RE and joint Entity and Relation Extraction (ERE) (first). The few-shot prompt templates for NER (second), RE (third), and Joint ERE (fourth). Different colors indicate different prompt design elements: gray for annotation guideline-based task instructions ${I}$, blue for retrieved demonstrations ${D}$, orange denotes the test example input $x_{test}$, and the green represents the expected output of test example output, which will be omitted during testing. $y_{test}$. Due to space constraints, we shortened the text of our prompts.
• Figure 3: Ablation study for the effectiveness of using annotation guideline to improve the Instruction ${I}$. "NER w/ Tag" denotes the performance gain with additional HTML tag setting.
• Figure 4: Performance trends of PL-Marker trained on varying number of documents for NER, end-to-end RE (Rel and Rel+) and RE.
• Figure 5: Annotation interface.