SituatedQA: Incorporating Extra-Linguistic Contexts into QA

TL;DR

This work introduces SituatedQA, a dataset and framework for open-retrieval QA where answers depend on extra-linguistic contexts like time and place. It formalizes definitions and two core tasks—context-aware QA and context-dependence detection—and builds a large-scale, crowdsourced dataset by identifying context-dependent questions from existing QA corpora and collecting temporally and geographically adapted answers. Through extensive experiments with retrieval-based and closed-book baselines, the authors demonstrate that current models struggle to adapt to present-day or locale-specific facts, show biases in geographic context, and benefit from explicit context-aware training and query augmentation. The study argues for incorporating extra-linguistic contexts into open-retrieval QA benchmarks to ensure global relevance and practical usefulness, and it provides data, code, and analyses to catalyze future research in dynamic, context-sensitive QA.

Abstract

Answers to the same question may change depending on the extra-linguistic contexts (when and where the question was asked). To study this challenge, we introduce SituatedQA, an open-retrieval QA dataset where systems must produce the correct answer to a question given the temporal or geographical context. To construct SituatedQA, we first identify such questions in existing QA datasets. We find that a significant proportion of information seeking questions have context-dependent answers (e.g., roughly 16.5% of NQ-Open). For such context-dependent questions, we then crowdsource alternative contexts and their corresponding answers. Our study shows that existing models struggle with producing answers that are frequently updated or from uncommon locations. We further quantify how existing models, which are trained on data collected in the past, fail to generalize to answering questions asked in the present, even when provided with an updated evidence corpus (a roughly 15 point drop in accuracy). Our analysis suggests that open-retrieval QA benchmarks should incorporate extra-linguistic context to stay relevant globally and in the future. Our data, code, and datasheet are available at https://situatedqa.github.io/ .

Figures (8)

• Figure 1: Examples of questions with answers that change depending on the temporal or geographical context.
• Figure 2: Data collection pipeline: Crowdworkers are first asked to identify context dependent questions. We then collect brief answer timelines for temporally dependent questions and location/answer pairs geographically dependent questions, each of which is then verified by another worker.
• Figure 3: The distribution of temporally dependent questions by the duration its previous answer was true for with examples.
• Figure 4: Interface for identifying temporally and geographically dependent questions.
• Figure 5: Interface for collecting geographical {Context / Answer} pairs. Annotators are also asked to verify the original location and answer.