LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledge

TL;DR

LiveSearchBench tackles the mismatch between static QA benchmarks and the dynamic nature of world knowledge by introducing a fully automated pipeline that constructs temporally grounded, SPARQL-verifiable QA instances from Wikidata deltas. The approach yields three difficulty levels (L1-L3) and validates each instance against the most recent snapshot to ensure a unique answer, enabling robust evaluation of retrieval and reasoning under evolving knowledge. Experiments reveal a significant recency gap: models struggle on post-pretraining facts, especially for multi-hop queries, and retrieval-augmented methods only partially mitigate this gap, with model scale offering limited relief. The work provides a scalable, continual benchmark resource that promotes evaluation designs emphasizing up-to-date evidence and coordinated search-reasoning capabilities.

Abstract

Evaluating large language models (LLMs) on question answering often relies on static benchmarks that reward memorization and understate the role of retrieval, failing to capture the dynamic nature of world knowledge. We present LiveSearchBench, an automated pipeline for constructing retrieval-dependent benchmarks from recent knowledge updates. Our method computes deltas between successive Wikidata snapshots, filters candidate triples for quality, and synthesizes natural-language questions at three levels of reasoning difficulty, each guaranteed to admit a unique, verifiable answer through SPARQL validation. The pipeline is fully automated, scalable across time, and minimizes human intervention, enabling continual regeneration of temporally grounded benchmarks. Experiments show a pronounced performance drop when models confront facts that post-date pretraining, with the gap most salient on multi-hop queries. Retrieval augmented methods and larger, instruction-tuned models provide partial gains but fail to close this recency gap. By design, LiveSearchBench shifts evaluation from static memorization toward tasks that require up-to-date retrieval and reasoning, offering a foundation for systematic, long-term assessment of LLMs under evolving knowledge.

Figures (6)

• Figure 1: A timeline of major QA benchmarks and model releases. The figure illustrates the historical reliance on static benchmarks, motivating the need for dynamic evaluation resources.
• Figure 2: Accuracy difference $\Delta_k=\text{Pass@}k_{\text{no-search}}-\text{Pass@}1_{\text{search}}$ across six QA benchmarks and multiple model sizes. Red regions denote that parametric-only inference outperforms retrieval@1, while blue regions indicate the opposite.
• Figure 3: Overview of the generation pipeline. We compute a knowledge delta between two Wikidata snapshots to obtain new or updated subject–relation–object (SRO) triples. After relation and entity based filtering, candidate triples are used to synthesize questions at three difficulty tiers: (L1) single-hop, (L2) multi-constraint multi-hop, and (L3) multi-hop with attribute fuzzing. All questions are verified against the current snapshot via SPARQL to ensure correctness.
• Figure 4: Dataset statistics of LiveSearchBench. (a) Distribution of questions across difficulty tiers L1–L3. (b) Frequency of the most common relation types in synthesized triples. Together, these plots illustrate both the diversity of reasoning requirements and the breadth of relation coverage in our benchmark.
• Figure 5: Absolute ($\Delta$) and relative (%) improvements of retrieval-based methods over Direct Answer, averaged across models, on the 2021 vs. 2025 batches.