Temporal Leakage in Search-Engine Date-Filtered Web Retrieval: A Case Study from Retrospective Forecasting

TL;DR

Temporal leakage in date-filtered search undermines retrospective forecasting by letting post-cutoff information contaminate retrieved documents. The authors conduct a systematic audit of Google Search across 393 resolved questions and 38,879 URLs, using an LLM-based judge to quantify leakage and its impact on forecast accuracy, finding that post-cutoff content can drastically inflate performance (Brier scores drop from $0.244$ to as low as $0.108$ with leaky documents). They identify four leakage mechanisms—Direct Page Updates, Related Content Leakage, Absence-based Signals, and Unreliable Metadata—and document a decreasing leakage trend over time, though it remains pervasive. The work argues for stronger retrieval safeguards or evaluation on frozen, time-stamped web snapshots to ensure credible retrospective forecasting and generalize to other retrieval pipelines.

Abstract

Search-engine date filters are widely used to enforce pre-cutoff retrieval in retrospective evaluations of search-augmented forecasters. We show this approach is unreliable: auditing Google Search with a before: filter, 71% of questions return at least one page containing strong post-cutoff leakage, and for 41%, at least one page directly reveals the answer. Using a large language model (LLM), gpt-oss-120b, to forecast with these leaky documents, we demonstrate an inflated prediction accuracy (Brier score 0.108 vs. 0.242 with leak-free documents). We characterize common leakage mechanisms, including updated articles, related-content modules, unreliable metadata/timestamps, and absence-based signals, and argue that date-restricted search is insufficient for temporal evaluation. We recommend stronger retrieval safeguards or evaluation on frozen, time-stamped web snapshots to ensure credible retrospective forecasting.

Figures (6)

• Figure 1: Date-filtered search retrieves a page updated after the question open date, leaking post-cutoff evidence into the LLM’s forecast. This inflates apparent performance and invalidates retrospective evaluation.
• Figure 2: Percentage of pages containing post-cutoff information by question open year (2021–2025). The dataset did not contain questions opened in 2024.
• Figure 3: Confusion matrix of human-LLM score
• Figure 4: Distribution of maximum leakage score per question.
• Figure 5: Number of retrieved pages by leakage score.