Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension

Alon Talmor, Jonathan Berant

TL;DR

The study addresses the challenge of generalization and transfer across multiple reading-comprehension datasets. It systematically evaluates DocQA and BERT-based approaches, analyzes factors affecting cross-dataset generalization, and demonstrates that pre-training on multiple RC datasets substantially boosts transfer and reduces annotation costs. The authors introduce MultiQA, a BERT-based model trained on multiple datasets that achieves state-of-the-art results on several benchmarks and shows robustness to adversarial inputs. They also provide open-source infrastructure to facilitate large-scale RC experimentation and benchmarking. These findings highlight dataset size and diversity as key bottlenecks and offer practical guidance for building general-purpose RC systems.

Abstract

A large number of reading comprehension (RC) datasets has been created recently, but little analysis has been done on whether they generalize to one another, and the extent to which existing datasets can be leveraged for improving performance on new ones. In this paper, we conduct such an investigation over ten RC datasets, training on one or more source RC datasets, and evaluating generalization, as well as transfer to a target RC dataset. We analyze the factors that contribute to generalization, and show that training on a source RC dataset and transferring to a target dataset substantially improves performance, even in the presence of powerful contextual representations from BERT (Devlin et al., 2019). We also find that training on multiple source RC datasets leads to robust generalization and transfer, and can reduce the cost of example collection for a new RC dataset. Following our analysis, we propose MultiQA, a BERT-based model, trained on multiple RC datasets, which leads to state-of-the-art performance on five RC datasets. We share our infrastructure for the benefit of the research community.

MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension

TL;DR

The study addresses the challenge of generalization and transfer across multiple reading-comprehension datasets. It systematically evaluates DocQA and BERT-based approaches, analyzes factors affecting cross-dataset generalization, and demonstrates that pre-training on multiple RC datasets substantially boosts transfer and reduces annotation costs. The authors introduce MultiQA, a BERT-based model trained on multiple datasets that achieves state-of-the-art results on several benchmarks and shows robustness to adversarial inputs. They also provide open-source infrastructure to facilitate large-scale RC experimentation and benchmarking. These findings highlight dataset size and diversity as key bottlenecks and offer practical guidance for building general-purpose RC systems.

Abstract

A large number of reading comprehension (RC) datasets has been created recently, but little analysis has been done on whether they generalize to one another, and the extent to which existing datasets can be leveraged for improving performance on new ones. In this paper, we conduct such an investigation over ten RC datasets, training on one or more source RC datasets, and evaluating generalization, as well as transfer to a target RC dataset. We analyze the factors that contribute to generalization, and show that training on a source RC dataset and transferring to a target dataset substantially improves performance, even in the presence of powerful contextual representations from BERT (Devlin et al., 2019). We also find that training on multiple source RC datasets leads to robust generalization and transfer, and can reduce the cost of example collection for a new RC dataset. Following our analysis, we propose MultiQA, a BERT-based model, trained on multiple RC datasets, which leads to state-of-the-art performance on five RC datasets. We share our infrastructure for the benefit of the research community.

Paper Structure

This paper contains 19 sections, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Datasets
  3. Models
  4. Pre-processing
  5. DocQA
  6. BertQA
  7. Controlled Experiments
  8. Do models generalize to unseen datasets?
  9. Global structure
  10. Does generalization improve with more data?
  11. Does pre-training improve results on small datasets?
  12. How to choose the dataset to pre-train on?
  13. How much target data is needed?
  14. Does context augmentation improve performance?
  15. MultiQA
  16. ...and 4 more sections

Figures (2)

  • Figure 1: A 2D-visualization of the similarity between different datasets using the force-directed placement algorithm. We mark datasets that use web snippets as context with triangles, Wikipedia with circles, and Newswire with squares. We color multi-hop reasoning datasets in red, trivia datasets in blue, and factoid RC datasets in green.
  • Figure 2: Learning curves for the five large datasets (top is DocQA and bottom is BertQA). The x-axis corresponds to the number of examples from the target dataset, and the y-axis is EM. The orange curve refers to training on the target dataset only, and the blue curve refers to pre-training on 75K examples from the nearest source dataset and fine-tuning on the target dataset. The green curve is training on a fixed number of examples from all 5 large datasets without fine-tuning (MultiQA).