Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era

Zhengqing Yuan, Kaiwen Shi, Zheyuan Zhang, Lichao Sun, Nitesh V. Chawla, Yanfang Ye

TL;DR

This work provides the first scalable infrastructure for auditing citations in the LLM era and practical tools to improve the trustworthiness of scientific references and defines unified metrics for citation faithfulness and evidence alignment.

Abstract

Scientific research relies on accurate citation for attribution and integrity, yet large language models (LLMs) introduce a new risk: fabricated references that appear plausible but correspond to no real publications. Such hallucinated citations have already been observed in submissions and accepted papers at major machine learning venues, exposing vulnerabilities in peer review. Meanwhile, rapidly growing reference lists make manual verification impractical, and existing automated tools remain fragile to noisy and heterogeneous citation formats and lack standardized evaluation. We present the first comprehensive benchmark and detection framework for hallucinated citations in scientific writing. Our multi-agent verification pipeline decomposes citation checking into claim extraction, evidence retrieval, passage matching, reasoning, and calibrated judgment to assess whether a cited source truly supports its claim. We construct a large-scale human-validated dataset across domains and define unified metrics for citation faithfulness and evidence alignment. Experiments with state-of-the-art LLMs reveal substantial citation errors and show that our framework significantly outperforms prior methods in both accuracy and interpretability. This work provides the first scalable infrastructure for auditing citations in the LLM era and practical tools to improve the trustworthiness of scientific references.

CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era

TL;DR

This work provides the first scalable infrastructure for auditing citations in the LLM era and practical tools to improve the trustworthiness of scientific references and defines unified metrics for citation faithfulness and evidence alignment.

Abstract

Scientific research relies on accurate citation for attribution and integrity, yet large language models (LLMs) introduce a new risk: fabricated references that appear plausible but correspond to no real publications. Such hallucinated citations have already been observed in submissions and accepted papers at major machine learning venues, exposing vulnerabilities in peer review. Meanwhile, rapidly growing reference lists make manual verification impractical, and existing automated tools remain fragile to noisy and heterogeneous citation formats and lack standardized evaluation. We present the first comprehensive benchmark and detection framework for hallucinated citations in scientific writing. Our multi-agent verification pipeline decomposes citation checking into claim extraction, evidence retrieval, passage matching, reasoning, and calibrated judgment to assess whether a cited source truly supports its claim. We construct a large-scale human-validated dataset across domains and define unified metrics for citation faithfulness and evidence alignment. Experiments with state-of-the-art LLMs reveal substantial citation errors and show that our framework significantly outperforms prior methods in both accuracy and interpretability. This work provides the first scalable infrastructure for auditing citations in the LLM era and practical tools to improve the trustworthiness of scientific references.
Paper Structure (45 sections, 4 equations, 4 figures, 5 tables)

This paper contains 45 sections, 4 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Reference Hallucination Detection
  4. Web Search Agent and Fact Checking
  5. Benchmark
  6. Real World Data Collection
  7. Human-Synthesized Data Generation
  8. Data Annotation
  9. Methodology
  10. Problem Formulation
  11. Collaborative Multi-Agent Pipeline
  12. Collaborative Multi-Agent Pipeline and Planning Model
  13. Experiments
  14. Experimental Setup
  15. Evaluation on Generated Benchmark
  16. ...and 30 more sections

Figures (4)

  • Figure 1: Motivation overview of our citation hallucination benchmark: existing gaps in closed-source citation checking tools and the unified, reliable evaluation framework enabled by our benchmark.
  • Figure 2: Taxonomy of citation hallucination types.
  • Figure 3: Overview of CiteAudit, a SOP-driven multi-agent citation verification framework.
  • Figure 4: Case studies of citation verification