Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The 17% Gap: Quantifying Epistemic Decay in AI-Assisted Survey Papers

H. Kemal İlter

TL;DR

The paper quantifies epistemic decay in AI-assisted survey literature by auditing 5,514 citations across 50 AI survey papers and applying a multi-stage forensic pipeline that distinguishes Sloppiness from Phantoms. It reports a persistent $P=0.17$ phantom rate, dominated by parsing artifacts ($78.5\%$) rather than pure hallucinations, and finds no strong temporal trend, indicating a stable equilibrium of decay. The authors formalize a Muller's Ratchet-like model to describe irreversible degradation and discuss methods to repair the citation graph via automated DOI verification and robust metadata grounding. The findings highlight a systemic vulnerability in the scientific record when AI tools handle formatting and metadata, urging infrastructure- and policy-level interventions to restore verifiability and reproducibility.

Abstract

The adoption of Large Language Models (LLMs) in scientific writing promises efficiency but risks introducing informational entropy. While "hallucinated papers" are a known artifact, the systematic degradation of valid citation chains remains unquantified. We conducted a forensic audit of 50 recent survey papers in Artificial Intelligence (N=5,514 citations) published between September 2024 and January 2026. We utilized a hybrid verification pipeline combining DOI resolution, Crossref metadata analysis, Semantic Scholar queries, and fuzzy text matching to distinguish between formatting errors ("Sloppiness") and verifiable non-existence ("Phantoms). We detect a persistent 17.0% Phantom Rate -- citations that cannot be resolved to any digital object despite aggressive forensic recovery. Diagnostic categorization reveals three distinct failure modes: pure hallucinations (5.1%), hallucinated identifiers with valid titles (16.4%), and parsing-induced matching failures (78.5%). Longitudinal analysis reveals a flat trend (+0.07 pp/month), suggesting that high-entropy citation practices have stabilized as an endemic feature of the field. The scientific citation graph in AI survey literature exhibits "link rot" at scale. This suggests a mechanism where AI tools act as "lazy research assistants," retrieving correct titles but hallucinating metadata, thereby severing the digital chain of custody required for reproducible science.

The 17% Gap: Quantifying Epistemic Decay in AI-Assisted Survey Papers

TL;DR

The paper quantifies epistemic decay in AI-assisted survey literature by auditing 5,514 citations across 50 AI survey papers and applying a multi-stage forensic pipeline that distinguishes Sloppiness from Phantoms. It reports a persistent phantom rate, dominated by parsing artifacts () rather than pure hallucinations, and finds no strong temporal trend, indicating a stable equilibrium of decay. The authors formalize a Muller's Ratchet-like model to describe irreversible degradation and discuss methods to repair the citation graph via automated DOI verification and robust metadata grounding. The findings highlight a systemic vulnerability in the scientific record when AI tools handle formatting and metadata, urging infrastructure- and policy-level interventions to restore verifiability and reproducibility.

Abstract

The adoption of Large Language Models (LLMs) in scientific writing promises efficiency but risks introducing informational entropy. While "hallucinated papers" are a known artifact, the systematic degradation of valid citation chains remains unquantified. We conducted a forensic audit of 50 recent survey papers in Artificial Intelligence (N=5,514 citations) published between September 2024 and January 2026. We utilized a hybrid verification pipeline combining DOI resolution, Crossref metadata analysis, Semantic Scholar queries, and fuzzy text matching to distinguish between formatting errors ("Sloppiness") and verifiable non-existence ("Phantoms). We detect a persistent 17.0% Phantom Rate -- citations that cannot be resolved to any digital object despite aggressive forensic recovery. Diagnostic categorization reveals three distinct failure modes: pure hallucinations (5.1%), hallucinated identifiers with valid titles (16.4%), and parsing-induced matching failures (78.5%). Longitudinal analysis reveals a flat trend (+0.07 pp/month), suggesting that high-entropy citation practices have stabilized as an endemic feature of the field. The scientific citation graph in AI survey literature exhibits "link rot" at scale. This suggests a mechanism where AI tools act as "lazy research assistants," retrieving correct titles but hallucinating metadata, thereby severing the digital chain of custody required for reproducible science.
Paper Structure (35 sections, 1 theorem, 22 equations, 3 figures, 5 tables, 1 algorithm)

This paper contains 35 sections, 1 theorem, 22 equations, 3 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Hallucination in Academic Contexts
  4. Citation Analysis and Link Rot
  5. The DOI System and Metadata Integrity
  6. Methodology
  7. Data Selection and Corpus Construction
  8. The Forensic Verification Pipeline
  9. Stage 1: Identifier Extraction
  10. Stage 2: Direct Resolution
  11. Stage 3: Entropy Filter
  12. Stage 4: Fuzzy Title Matching
  13. Stage 5: Classification
  14. Phantom Diagnostic Taxonomy
  15. Statistical Analysis
  16. ...and 20 more sections

Key Result

Proposition 1

With phantom rate $p = 0.17$, the half-life of citation integrity (time for $G_t = 0.5 G_0$) is:

Figures (3)

  • Figure 1: Phantom citation rate over time (September 2024 -- January 2026). Each point represents one paper; point size proportional to citation count. The dashed trend line shows negligible slope ($\hat{\beta}_1 = +0.07$ pp/month, $R^2 = 0.003$), indicating a stable equilibrium of decay. Mean phantom rate $\bar{P} = 16.5\%$, $\sigma_P = 14.1\%$. The bottom panel shows monthly citation breakdown by verification status.
  • Figure 2: Diagnostic categorization of phantom citations ($N=939$). The donut chart shows three failure modes classified by Equation \ref{['eq:phantom_taxonomy']}: Syntax Error (78.5%, $s^* \geq 25\%$), Broken Link (16.4%, DOI $\to$ 404), and Ghost (5.1%, $s^* < 25\%$). The dominance of parsing-related failures suggests that most "phantoms" are potentially recoverable with improved text extraction.
  • Figure 3: Top 15 papers ranked by phantom citation rate $P_i$. Horizontal bars colored by phantom rate (green = low, red = high). Maximum observed rate = 58.8%. The coefficient of variation $CV = 0.85$ indicates high inter-paper dispersion.

Theorems & Definitions (4)

  • Definition 1: Space Ratio Entropy Filter
  • Definition 2: Levenshtein Similarity Ratio
  • Definition 3: Citation Decay Model
  • Proposition 1: Half-Life of Citation Integrity