MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge

TL;DR

MCScript introduces a large narrative dataset designed to assess machine comprehension that requires script knowledge and commonsense reasoning about everyday activities. By crowdsourcing texts, questions, and answer choices around script scenarios rather than individual texts, the dataset emphasizes non-literal inferences and extrinsic evaluation of script knowledge. The authors provide thorough data collection, validation, and statistics, and show that while neural models outperform simple baselines, the task remains challenging, especially for yes/no and non-literal questions. The resource supports a SemEval-2018 shared task and aims to push progress in robust commonsense reasoning for NLP systems.

Abstract

We introduce a large dataset of narrative texts and questions about these texts, intended to be used in a machine comprehension task that requires reasoning using commonsense knowledge. Our dataset complements similar datasets in that we focus on stories about everyday activities, such as going to the movies or working in the garden, and that the questions require commonsense knowledge, or more specifically, script knowledge, to be answered. We show that our mode of data collection via crowdsourcing results in a substantial amount of such inference questions. The dataset forms the basis of a shared task on commonsense and script knowledge organized at SemEval 2018 and provides challenging test cases for the broader natural language understanding community.

Figures (4)

• Figure 1: An example for a text snippet with two reading comprehension questions.
• Figure 2: Distribution of question types in the data.
• Figure 3: An example text with 2 questions from MCScript
• Figure 4: Accuracy values of the baseline models on question types appearing $>25$ times.