StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models

TL;DR

StreamingQA introduces a large-scale, temporally grounded QA benchmark that pairs time-stamped questions with a chronologically ordered news corpus to study knowledge adaptation over time. By evaluating both closed-book (parametric) and open-book (semi-parametric) QA models under streaming knowledge updates, the paper reveals that iterative LM fine-tuning improves adaptation while full retraining yields the strongest performance, and that updating the retrieval index is the primary driver for fast open-book adaptation. The work uncovers a complementary dynamic: parametric updates excel for frequent entities, whereas retrieval-based approaches better handle less frequent knowledge, informing how to balance updating the model weights versus the knowledge store. Additional analyses on temporal lag, timing specifications, and static vs streaming settings demonstrate practical pathways to maintain up-to-date QA systems in a changing information landscape. Overall, StreamingQA provides a realistic evaluation framework and actionable insights for continual knowledge integration in QA models, with implications for deployment in dynamic, real-world settings.

Abstract

Knowledge and language understanding of models evaluated through question answering (QA) has been usually studied on static snapshots of knowledge, like Wikipedia. However, our world is dynamic, evolves over time, and our models' knowledge becomes outdated. To study how semi-parametric QA models and their underlying parametric language models (LMs) adapt to evolving knowledge, we construct a new large-scale dataset, StreamingQA, with human written and generated questions asked on a given date, to be answered from 14 years of time-stamped news articles. We evaluate our models quarterly as they read new articles not seen in pre-training. We show that parametric models can be updated without full retraining, while avoiding catastrophic forgetting. For semi-parametric models, adding new articles into the search space allows for rapid adaptation, however, models with an outdated underlying LM under-perform those with a retrained LM. For questions about higher-frequency named entities, parametric updates are particularly beneficial. In our dynamic world, the StreamingQA dataset enables a more realistic evaluation of QA models, and our experiments highlight several promising directions for future research.

Figures (17)

• Figure 1: The StreamingQA task: we emulate a realistic scenario where a QA system needs to respond to user questions about a mix of recent and past events.
• Figure 2: Details of the StreamingQA dataset.
• Figure 3: Left: F1 score on the whole evaluation dataset of CB$_{\text{+Stale}}$, CB$_{\text{+Retr.}}$, and CB$_{\text{+FT}}$ fine-tuned on articles published until the specified cut-off dates. Right: The effect of a temporal lag between the final training month of CB$_{\text{+FT}}$ and question dates for generated questions, relative to CB$_{\text{+Retr.}}$.
• Figure 4: Adaptation and forgetting on recent subsets (generated, left; written, right). We observe that adapting the generator helps the FiD model, and helps the OB model when fully retrained, compared to index update only. Open-book models allow for much faster adaptation to recent knowledge than closed-book, with almost no forgetting. (IU = search index updated, FT = fine-tuned LM)
• Figure 5: Adaptation and forgetting on past generated questions. We see only a slight improvement as the model acquires knowledge about 2020. We do not observe forgetting.