When Facts Change: Probing LLMs on Evolving Knowledge with evolveQA

TL;DR

This work tackles the challenge that LLMs often retain outdated information as knowledge evolves. It introduces evolveQA, a benchmark built from time-stamped AWS, Azure, and WHO corpora, employing a multi-stage framework to identify evolving facts and generate time-aware QA across open-ended, MCQ, and verifiable formats. Across 12 LLMs and three probing formats, the study finds consistent performance degradation on evolving knowledge (6–31%), with open-ended questions most affected, though MCQ and verifiable formats partially mitigate the losses. The benchmark provides a realistic, scalable testbed for developing temporally robust LLMs and highlights actionable directions for addressing temporal knowledge conflicts in high-stakes domains.

Abstract

LLMs often fail to handle temporal knowledge conflicts--contradictions arising when facts evolve over time within their training data. Existing studies evaluate this phenomenon through benchmarks built on structured knowledge bases like Wikidata, but they focus on widely-covered, easily-memorized popular entities and lack the dynamic structure needed to fairly evaluate LLMs with different knowledge cut-off dates. We introduce evolveQA, a benchmark specifically designed to evaluate LLMs on temporally evolving knowledge, constructed from 3 real-world, time-stamped corpora: AWS updates, Azure changes, and WHO disease outbreak reports. Our framework identifies naturally occurring knowledge evolution and generates questions with gold answers tailored to different LLM knowledge cut-off dates. Through extensive evaluation of 12 open and closed-source LLMs across 3 knowledge probing formats, we demonstrate significant performance drops of up to 31% on evolveQA compared to static knowledge questions.

Figures (10)

• Figure 1: LLMs often give outdated answers to open-ended questions, even though they can identify correct and current information when given multiple-choice options; this suggests that while LLMs hold more recent knowledge in their parameters, they struggle to recall it without explicit prompting.
• Figure 2: Framework for evolveQA construction. It operates in two phases: 1. Evolving Fact Identification: (1) Extract entities and concepts, (2) cluster concepts into topics, (3) obtain documents for each {Entity, Concept} pair, (4) cluster document chunks to identify temporal attributes. 2. Grounded QA Curation: (5) Generate specific questions and (6) curate time-sensitive gold answers for each {Entity, Concept, Attribute} tuple.
• Figure 3: Performance comparison between evolveQA and control set on open-ended QA, aggregated across the three domains. We observe a significant performance degredation on evolveQA across all models.
• Figure 4: Shows two examples where Claude-3.5-Haiku generates correct response in open-ended QA whereas, it generates an incorrect response for its corresponding MCQ format in the control set.
• Figure 5: Shows the prompt used for concept and salient entity extraction.