Can LLMs Augment Low-Resource Reading Comprehension Datasets? Opportunities and Challenges

TL;DR

The paper investigates whether GPT-4 can augment low-resource extractive reading comprehension datasets by generating synthetic contexts and QA pairs via a two-stage pipeline (Context Generation, QA Generation) and a Round Trip Filtration mechanism. It demonstrates that, in three real-world domains (CovidQA, PolicyQA, TechQA), synthetic data can improve downstream MRC performance in some settings—especially with one-shot prompts on CovidQA and PolicyQA—and even achieve state-of-the-art results on CovidQA, while failing to yield gains on ultra-small TechQA data. The authors provide a reproducible workflow and release augmented datasets, highlighting both the substantial opportunities and practical challenges of using LLM-driven synthetic data for QA in low-resource regimes. Overall, the study shows that LLM-based synthetic data has notable potential to alleviate data scarcity, but success depends on dataset size, domain characteristics, and effective quality-filtering and knowledge integration strategies.

Abstract

Large Language Models (LLMs) have demonstrated impressive zero shot performance on a wide range of NLP tasks, demonstrating the ability to reason and apply commonsense. A relevant application is to use them for creating high quality synthetic datasets for downstream tasks. In this work, we probe whether GPT-4 can be used to augment existing extractive reading comprehension datasets. Automating data annotation processes has the potential to save large amounts of time, money and effort that goes into manually labelling datasets. In this paper, we evaluate the performance of GPT-4 as a replacement for human annotators for low resource reading comprehension tasks, by comparing performance after fine tuning, and the cost associated with annotation. This work serves to be the first analysis of LLMs as synthetic data augmenters for QA systems, highlighting the unique opportunities and challenges. Additionally, we release augmented versions of low resource datasets, that will allow the research community to create further benchmarks for evaluation of generated datasets.

Figures (1)

• Figure 1: Overview of our methodology using PolicyQA as an example with 2-shot prompts.