Large-scale Cloze Test Dataset Created by Teachers

TL;DR

CLOTH addresses the gap between automatically generated cloze datasets and human reading comprehension by introducing a large-scale, teacher-crafted cloze dataset from English exams. The study shows humans markedly outperform state-of-the-art language models, with long-range context being the primary bottleneck, and demonstrates that human-created data present greater challenge than automatically-generated data. It also introduces a representativeness-based training approach to combine human-created and generated data, improving performance and offering a path for leveraging mixed data sources in language understanding tasks. Overall, CLOTH provides a valuable benchmark for advancing long-context modeling in both language modeling and machine comprehension research.

Abstract

Cloze tests are widely adopted in language exams to evaluate students' language proficiency. In this paper, we propose the first large-scale human-created cloze test dataset CLOTH, containing questions used in middle-school and high-school language exams. With missing blanks carefully created by teachers and candidate choices purposely designed to be nuanced, CLOTH requires a deeper language understanding and a wider attention span than previously automatically-generated cloze datasets. We test the performance of dedicatedly designed baseline models including a language model trained on the One Billion Word Corpus and show humans outperform them by a significant margin. We investigate the source of the performance gap, trace model deficiencies to some distinct properties of CLOTH, and identify the limited ability of comprehending the long-term context to be the key bottleneck.

Figures (2)

• Figure 1: Representativeness prediction for each word. Lighter color means less representative. The words deleted by human as blanks are in bold text.
• Figure 2: Model and human's performance on questions with different types. Our model will be introduced in Sec. \ref{['sec:combine']}.