Dataset Mention Extraction in Scientific Articles Using Bi-LSTM-CRF Model

TL;DR

This work addresses the problem of automatically extracting dataset mentions from scientific articles to improve credit and tracking of dataset usage. It proposes a Bi-LSTM-CRF sequence labeling model that incorporates character-level embeddings, pre-trained word vectors, and a CRF decoder to detect dataset mentions, evaluated on the Rich Context Dataset where the best model achieves $F_{1}=0.885$. The study highlights challenges in the data, such as nested mentions and noisy linkages between mentions and datasets, and discusses limitations and potential improvements, including attention mechanisms and Transformer architectures. The findings suggest that neural sequence labeling can effectively identify dataset mentions and lays groundwork for better dataset citation practices and usage tracking across scientific literature.

Abstract

Datasets are critical for scientific research, playing an important role in replication, reproducibility, and efficiency. Researchers have recently shown that datasets are becoming more important for science to function properly, even serving as artifacts of study themselves. However, citing datasets is not a common or standard practice in spite of recent efforts by data repositories and funding agencies. This greatly affects our ability to track their usage and importance. A potential solution to this problem is to automatically extract dataset mentions from scientific articles. In this work, we propose to achieve such extraction by using a neural network based on a Bi-LSTM-CRF architecture. Our method achieves F1 = 0.885 in social science articles released as part of the Rich Context Dataset. We discuss the limitations of the current datasets and propose modifications to the model to be done in the future.

Figures (1)

• Figure 1: Network Architecture of Bi-LSTM-CRF network