Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models

TL;DR

This paper introduces EvolvingQA, a temporally evolving QA benchmark that evaluates lifelong language models on up-to-date and updated knowledge drawn from Wikipedia snapshots. It combines an automated continual pre-training pipeline on Changed sets with a QA-driven evaluation across Unchanged, New, and Edited subsets to measure forgetting, acquisition, and updating of knowledge. Through experiments with multiple baselines (Initial, Full, K-Adapter, LoRA, DPR), the study reveals persistent challenges in forgetting outdated information and updating numerical or temporal facts, attributed in part to small gradient updates during continued training. The findings highlight the insufficiency of purely retrieval-based or static training approaches for real-world temporal knowledge and underscore the need for methods that more effectively propagate updates while preserving prior knowledge, with implications for open-domain QA systems and long-lived AI deployments.

Abstract

The dynamic nature of knowledge in an ever-changing world presents challenges for language models trained on static data; the model in the real world often requires not only acquiring new knowledge but also overwriting outdated information into updated ones. To study the ability of language models for these time-dependent dynamics in human language, we introduce a novel task, EvolvingQA, a temporally evolving question-answering benchmark designed for training and evaluating LMs on an evolving Wikipedia database. The construction of EvolvingQA is automated with our pipeline using large language models. We uncover that existing continual learning baselines suffer from updating and removing outdated knowledge. Our analysis suggests that models fail to rectify knowledge due to small weight gradients. In addition, we elucidate that language models particularly struggle to reflect the change of numerical or temporal information. Our work aims to model the dynamic nature of real-world information, suggesting faithful evaluations of the evolution-adaptability of language models.

Figures (10)

• Figure 1: An overview of our evaluation benchmark, EvolvingQA. Our benchmark employs LLM to generate question-answer pairs based on the changes in Wikipedia's snapshots, effectively capturing the temporal evolution of the knowledge base.
• Figure 2: Construction pipeline of Edited. The final question-answers pair after filtering processes in this Figure is included in Edited06. The full description of the pipeline is in Section \ref{['subsec:benchmark']} and Appendix \ref{['appx:eval_detail']}.
• Figure 3: An example of the prompt we use in generating QA pairs in Edited set. The blue-colored messages are one-shot demonstration to make sure GPT-3.5 follow the instruction more accurately and generate question-answer instances in a desired format.
• Figure 4: The bar plot that shows the trend of F1 scores through continual learning of Changed sets. Note that a single Unchanged set is used to evaluate on all time steps.
• Figure 5: The scatter plot of samples in Changed03 according to the number of masked entities and gradient norm. Each dot indicates a sample from either New knowledge or Edited knowledge in Changed. The $x$-axis shows the number of masked entities in a sample. The $y$-axis shows the Frobenius norm of weight gradients of each sample.