MSciNLI: A Diverse Benchmark for Scientific Natural Language Inference
Mobashir Sadat, Cornelia Caragea
TL;DR
MSciNLI introduces a diverse, multi-domain scientific NLI benchmark with 132,320 sentence pairs drawn from five domains, enabling systematic study of domain shift in scientific inference. Training data are automatically labeled via distant supervision, while 4,000 test and 1,000 dev examples are manually annotated to ensure high-quality evaluation. Strong baselines include fine-tuned RoBERTa and SciBERT, as well as Llama-2 prompts, revealing that MSciNLI is challenging for both PLMs and current LLMs and that domain diversity improves transfer to downstream tasks through intermediate training. The work demonstrates robust domain-shift analysis, cross-dataset training benefits, and the potential of scientific NLI as an intermediate task for enhancing downstream scientific NLP applications, with code and data openly available on GitHub.
Abstract
The task of scientific Natural Language Inference (NLI) involves predicting the semantic relation between two sentences extracted from research articles. This task was recently proposed along with a new dataset called SciNLI derived from papers published in the computational linguistics domain. In this paper, we aim to introduce diversity in the scientific NLI task and present MSciNLI, a dataset containing 132,320 sentence pairs extracted from five new scientific domains. The availability of multiple domains makes it possible to study domain shift for scientific NLI. We establish strong baselines on MSciNLI by fine-tuning Pre-trained Language Models (PLMs) and prompting Large Language Models (LLMs). The highest Macro F1 scores of PLM and LLM baselines are 77.21% and 51.77%, respectively, illustrating that MSciNLI is challenging for both types of models. Furthermore, we show that domain shift degrades the performance of scientific NLI models which demonstrates the diverse characteristics of different domains in our dataset. Finally, we use both scientific NLI datasets in an intermediate task transfer learning setting and show that they can improve the performance of downstream tasks in the scientific domain. We make our dataset and code available on Github.