Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Benchmarks Age: Temporal Misalignment through Large Language Model Factuality Evaluation

Xunyi Jiang, Dingyi Chang, Julian McAuley, Xin Xu

TL;DR

Static factuality benchmarks quickly become stale as real-world information updates, risking unfair penalties for up-to-date LLMs. The authors present a retrieval-based pipeline to obtain current facts and three metrics (DDS, EMR, TAG) across five benchmarks and eight LLMs to quantify aging and its impact. Their findings show a large portion of time-sensitive items are outdated, and benchmark aging can mislabel correct outputs, with newer LLMs often aligning more with real-world facts than with gold labels. The work provides a testbed and urges temporal-aware benchmarking to improve reliability of factuality evaluation.

Abstract

The rapid evolution of large language models (LLMs) and the real world has outpaced the static nature of widely used evaluation benchmarks, raising concerns about their reliability for evaluating LLM factuality. While substantial works continue to rely on the popular but old benchmarks, their temporal misalignment with real-world facts and modern LLMs, and their effects on LLM factuality evaluation remain underexplored. Therefore, in this work, we present a systematic investigation of this issue by examining five popular factuality benchmarks and eight LLMs released across different years. An up-to-date fact retrieval pipeline and three metrics are tailored to quantify benchmark aging and its impact on LLM factuality evaluation. Experimental results and analysis illustrate that a considerable portion of samples in the widely used factuality benchmarks are outdated, leading to unreliable assessments of LLM factuality. We hope our work can provide a testbed to assess the reliability of a benchmark for LLM factuality evaluation and inspire more research on the benchmark aging issue. Codes are available in https://github.com/JiangXunyi/BenchAge.

When Benchmarks Age: Temporal Misalignment through Large Language Model Factuality Evaluation

TL;DR

Static factuality benchmarks quickly become stale as real-world information updates, risking unfair penalties for up-to-date LLMs. The authors present a retrieval-based pipeline to obtain current facts and three metrics (DDS, EMR, TAG) across five benchmarks and eight LLMs to quantify aging and its impact. Their findings show a large portion of time-sensitive items are outdated, and benchmark aging can mislabel correct outputs, with newer LLMs often aligning more with real-world facts than with gold labels. The work provides a testbed and urges temporal-aware benchmarking to improve reliability of factuality evaluation.

Abstract

The rapid evolution of large language models (LLMs) and the real world has outpaced the static nature of widely used evaluation benchmarks, raising concerns about their reliability for evaluating LLM factuality. While substantial works continue to rely on the popular but old benchmarks, their temporal misalignment with real-world facts and modern LLMs, and their effects on LLM factuality evaluation remain underexplored. Therefore, in this work, we present a systematic investigation of this issue by examining five popular factuality benchmarks and eight LLMs released across different years. An up-to-date fact retrieval pipeline and three metrics are tailored to quantify benchmark aging and its impact on LLM factuality evaluation. Experimental results and analysis illustrate that a considerable portion of samples in the widely used factuality benchmarks are outdated, leading to unreliable assessments of LLM factuality. We hope our work can provide a testbed to assess the reliability of a benchmark for LLM factuality evaluation and inspire more research on the benchmark aging issue. Codes are available in https://github.com/JiangXunyi/BenchAge.

Paper Structure

This paper contains 34 sections, 4 equations, 17 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Experimental Setup
  3. Time-sensitive Samples Extraction
  4. Latest Fact Retrieval
  5. Temporal Comparison
  6. RQ1
  7. RQ2
  8. Experimental Results and Analysis
  9. RQ1: A Considerable Portion of the Benchmarks Are Outdated
  10. RQ2: Benchmark Aging Affects the Reliability of LLM Evaluation
  11. The outdated benchmarks can mislabel factually correct model responses.
  12. The present LLMs are more aligned with real-world facts than with gold answers in the benchmarks.
  13. Dataset Analysis
  14. The usage of static benchmarks with outdated information is increasing.
  15. The outdated contexts amplify the temporal misalignment.
  16. ...and 19 more sections

Figures (17)

  • Figure 1: Experimental setups. We first extract time-sensitive samples and then collect the corresponding real-world fact (with the latest fact retrieval pipeline), LLM response, and the gold label in the benchmark for each sample. Finally, we apply the proposed metrics to measure the temporal misalignment among them.
  • Figure 2: $TAG$ across the LLMs and benchmarks.
  • Figure 3: Annual citation growth of the benchmark.
  • Figure 4: Prompt for Determining the Time-Sensitivity of Dataset Questions.
  • Figure 5: Instructions for Human Evaluation of Time-sensitive Questions
  • ...and 12 more figures