GooAQ: Open Question Answering with Diverse Answer Types

TL;DR

GooAQ tackles the lack of QA benchmarks with diverse answer types by compiling a large-scale, open QA dataset from Google autocomplete and answer boxes, enabling evaluation across short, snippet, and collection outputs. The authors benchmark T5 models in a closed-book setting, revealing that pre-training strongly benefits long-form and multi-item outputs, while labeled data chiefly aids short answers. They provide comprehensive quality assessments and show GooAQ transfers to ELI5, illustrating cross-task utility. The work establishes GooAQ as both a challenging testbed for diverse answer types and a valuable data source for training and evaluating long-form QA systems, with implications for building neural knowledge bases and robust question-generation pipelines.

Abstract

While day-to-day questions come with a variety of answer types, the current question-answering (QA) literature has failed to adequately address the answer diversity of questions. To this end, we present GooAQ, a large-scale dataset with a variety of answer types. This dataset contains over 5 million questions and 3 million answers collected from Google. GooAQ questions are collected semi-automatically from the Google search engine using its autocomplete feature. This results in naturalistic questions of practical interest that are nonetheless short and expressed using simple language. GooAQ answers are mined from Google's responses to our collected questions, specifically from the answer boxes in the search results. This yields a rich space of answer types, containing both textual answers (short and long) as well as more structured ones such as collections. We benchmarkT5 models on GooAQ and observe that: (a) in line with recent work, LM's strong performance on GooAQ's short-answer questions heavily benefit from annotated data; however, (b) their quality in generating coherent and accurate responses for questions requiring long responses (such as 'how' and 'why' questions) is less reliant on observing annotated data and mainly supported by their pre-training. We release GooAQ to facilitate further research on improving QA with diverse response types.

Figures (9)

• Figure 1: Examples from GooAQ showing different types of the questions considered in this study. Each input is a natural language question, mapped to textual answer(s). The questions/answers come with answer type which are inferred from meta information of the search results.
• Figure 2: Comparison of question length distributions
• Figure 3: The distribution of common bigrams in questions of GooAQ (a,b,c) vs. NQ (d).
• Figure 4: Evaluation of T5 (small,11B) models on different sub-tasks of GooAQ via automatic metrics (top) and human judgements (bottom). For human evaluation, 50% is the border at which the model output and the ground truth responses are indistinguishable. The short-answer sub-tasks ($\mathcal{T}_{short}$; left) have a relatively low performance when supervised with $2k$ instances. However, they benefit more than the long-answer sub-tasks ($\mathcal{T}_{snippet}$ & $\mathcal{T}_{collection}$) from more labeled data. Additionally, we observe that the gap between the two systems is bigger in terms of human evaluation (compared to the corresponding gap in terms of automatic evaluation), especially in the long response tasks (middle & right).
• Figure 5: Crowdsourcing interface used for human assessment of our baselines (§\ref{['sec:experiments']}). We use a similar template (with a single answer) to estimate the quality of GooAQ (§\ref{['sec:collection']}).