Improving Machine Reading Comprehension with General Reading Strategies
Kai Sun, Dian Yu, Dong Yu, Claire Cardie
TL;DR
This work tackles non-extractive machine reading comprehension under limited computational resources by introducing three cognitive-reading-inspired strategies: Back and Forth Reading (BF), Highlighting (HL), and Self-Assessment (SA). By fine-tuning a pre-trained transformer (GPT) on RACE with these strategies and ensembling, the authors achieve a $5.8$ percentage point improvement over prior best results on RACE, and further improve six representative non-extractive MRC datasets (ARC, OpenBookQA, MCTest, SemEval-2018 Task 11, ROCStories, MultiRC) by $6.2$ percentage points on average. SA generates unsupervised practice questions from text, HL emphasizes content words via trainable highlight embeddings, and BF exploits input-order variations to capture cross-text relations; together they demonstrate strong transfer when adapting to target tasks. The results indicate that well-designed, domain-independent fine-tuning strategies can substantially enhance MRC performance with limited labeled data, offering practical, generalizable benefits and a reproducible codebase for further research.
Abstract
Reading strategies have been shown to improve comprehension levels, especially for readers lacking adequate prior knowledge. Just as the process of knowledge accumulation is time-consuming for human readers, it is resource-demanding to impart rich general domain knowledge into a deep language model via pre-training. Inspired by reading strategies identified in cognitive science, and given limited computational resources -- just a pre-trained model and a fixed number of training instances -- we propose three general strategies aimed to improve non-extractive machine reading comprehension (MRC): (i) BACK AND FORTH READING that considers both the original and reverse order of an input sequence, (ii) HIGHLIGHTING, which adds a trainable embedding to the text embedding of tokens that are relevant to the question and candidate answers, and (iii) SELF-ASSESSMENT that generates practice questions and candidate answers directly from the text in an unsupervised manner. By fine-tuning a pre-trained language model (Radford et al., 2018) with our proposed strategies on the largest general domain multiple-choice MRC dataset RACE, we obtain a 5.8% absolute increase in accuracy over the previous best result achieved by the same pre-trained model fine-tuned on RACE without the use of strategies. We further fine-tune the resulting model on a target MRC task, leading to an absolute improvement of 6.2% in average accuracy over previous state-of-the-art approaches on six representative non-extractive MRC datasets from different domains (i.e., ARC, OpenBookQA, MCTest, SemEval-2018 Task 11, ROCStories, and MultiRC). These results demonstrate the effectiveness of our proposed strategies and the versatility and general applicability of our fine-tuned models that incorporate these strategies. Core code is available at https://github.com/nlpdata/strategy/.