AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge

TL;DR

This paper tackles data contamination in LLM evaluation by static benchmarks inflating performance and proposes AntiLeakBench, a framework that constructs contamination-free QA samples from knowledge updated after model cutoff times and attaches verifiable Wikipedia-based supporting documents. The method identifies updated knowledge via Wikidata claims (post-cutoff updates) and builds an automated workflow that requires no human labor, supporting multilingual evaluation. It demonstrates that pre-cutoff data often contaminate assessments while post-cutoff samples provide a stricter, more reliable measure of understanding and reasoning, with experiments across 12 open models and 2 proprietary models showing substantial gaps and the robustness of contamination-free evaluation. The work has practical impact by enabling scalable, fair benchmarking aligned with real-world knowledge dynamics, and highlights the need for ongoing, automated maintenance of evaluation suites as knowledge evolves.

Abstract

Data contamination hinders fair LLM evaluation by introducing test data into newer models' training sets. Existing studies solve this challenge by updating benchmarks with newly collected data. However, they fail to guarantee contamination-free evaluation as the newly collected data may contain pre-existing knowledge, and their benchmark updates rely on intensive human labor. To address these issues, we in this paper propose AntiLeak-Bench, an automated anti-leakage benchmarking framework. Instead of simply using newly collected data, we construct samples with explicitly new knowledge absent from LLMs' training sets, which thus ensures strictly contamination-free evaluation. We further design a fully automated workflow to build and update our benchmark without human labor. This significantly reduces the cost of benchmark maintenance to accommodate emerging LLMs. Through extensive experiments, we highlight that data contamination likely exists before LLMs' cutoff time and demonstrate AntiLeak-Bench effectively overcomes this challenge.

Figures (4)

• Figure 1: Illustration of AntiLeakBench. It constructs contamination-free samples based on the knowledge updated after LLMs' cutoff time, which thus are not in LLMs' training sets.
• Figure 2: Illustration of the automated benchmark building workflow without human labor. After data preparation, it includes three main steps: (1) Identify updated knowledge after the cutoff time; (2) Build supporting documents; (3) Construct contamination-free samples (\ref{['fig_multihop']} exemplifies how to construct multi-hop samples).
• Figure 4: EM and F1 performance at each time interval. Marker $\;\;$ denotes LLM's cutoff time.
• Figure 5: Correct and outdated option proportions at each time interval. Marker $\;\;$ denotes LLM's cutoff time.