Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Simple Yet Strong Pipeline for HotpotQA

Dirk Groeneveld, Tushar Khot, Mausam, Ashish Sabharwal

TL;DR

The paper challenges the notion that state-of-the-art multi-hop QA requires complex architectures by introducing Quark, a simple three-stage pipeline that independently scores sentences, uses a span-prediction model on selected sentences, and then identifies supporting sentences conditioned on the predicted answer. Across distractor and fullwiki HotpotQA settings, Quark matches or exceeds prior BERT-based methods and nears RoBERTa-level performance, highlighting the value of simple, well-tuned designs. Ablation studies reveal the importance of paragraph context and a second pass conditioned on the answer, while results suggest that retrieval in HotpotQA does not inherently require multi-hop reasoning. The work advocates prioritizing simpler baselines before escalating model complexity and encourages evaluation on other datasets to validate the necessity of advanced techniques.

Abstract

State-of-the-art models for multi-hop question answering typically augment large-scale language models like BERT with additional, intuitively useful capabilities such as named entity recognition, graph-based reasoning, and question decomposition. However, does their strong performance on popular multi-hop datasets really justify this added design complexity? Our results suggest that the answer may be no, because even our simple pipeline based on BERT, named Quark, performs surprisingly well. Specifically, on HotpotQA, Quark outperforms these models on both question answering and support identification (and achieves performance very close to a RoBERTa model). Our pipeline has three steps: 1) use BERT to identify potentially relevant sentences independently of each other; 2) feed the set of selected sentences as context into a standard BERT span prediction model to choose an answer; and 3) use the sentence selection model, now with the chosen answer, to produce supporting sentences. The strong performance of Quark resurfaces the importance of carefully exploring simple model designs before using popular benchmarks to justify the value of complex techniques.

A Simple Yet Strong Pipeline for HotpotQA

TL;DR

The paper challenges the notion that state-of-the-art multi-hop QA requires complex architectures by introducing Quark, a simple three-stage pipeline that independently scores sentences, uses a span-prediction model on selected sentences, and then identifies supporting sentences conditioned on the predicted answer. Across distractor and fullwiki HotpotQA settings, Quark matches or exceeds prior BERT-based methods and nears RoBERTa-level performance, highlighting the value of simple, well-tuned designs. Ablation studies reveal the importance of paragraph context and a second pass conditioned on the answer, while results suggest that retrieval in HotpotQA does not inherently require multi-hop reasoning. The work advocates prioritizing simpler baselines before escalating model complexity and encourages evaluation on other datasets to validate the necessity of advanced techniques.

Abstract

State-of-the-art models for multi-hop question answering typically augment large-scale language models like BERT with additional, intuitively useful capabilities such as named entity recognition, graph-based reasoning, and question decomposition. However, does their strong performance on popular multi-hop datasets really justify this added design complexity? Our results suggest that the answer may be no, because even our simple pipeline based on BERT, named Quark, performs surprisingly well. Specifically, on HotpotQA, Quark outperforms these models on both question answering and support identification (and achieves performance very close to a RoBERTa model). Our pipeline has three steps: 1) use BERT to identify potentially relevant sentences independently of each other; 2) feed the set of selected sentences as context into a standard BERT span prediction model to choose an answer; and 3) use the sentence selection model, now with the chosen answer, to produce supporting sentences. The strong performance of Quark resurfaces the importance of carefully exploring simple model designs before using popular benchmarks to justify the value of complex techniques.

Paper Structure

This paper contains 13 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Pipeline Model: Quark
  4. Sentence Scoring Module
  5. Question Answering Module
  6. Bringing it Together: Distractor Setting
  7. Bringing it Together: Fullwiki Setting
  8. Experiments
  9. Ablation
  10. Conclusion
  11. Appendix
  12. Training the sentence scoring model
  13. Training the span prediction model

Figures (1)

  • Figure 1: Overview of the Quark model, with a question and context paragraphs as input. In both blue boxes, sentences are scored independently from one another. $r_{na}(s)$ and $r_a(s)$ use the same model architecture with different weights.