Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hallucinations in Bibliographic Recommendation: Citation Frequency as a Proxy for Training Data Redundancy

Junichiro Niimi

TL;DR

This study investigates whether LLM hallucinations in bibliographic recommendations arise from training-data redundancy by using citation counts as a proxy. The authors generate 100 bibliographic records across 20 CS domains with GPT-4.1, validate them via Google Scholar, and measure factual fidelity with cosine similarity on metadata; they reveal domain-dependent hallucination rates and a strong positive link between citation frequency and factual accuracy. A non-linear memorization pattern emerges, with a threshold around $1000$ citations where highly cited papers are recalled verbatim, implying memorization rather than generalized generation. These findings suggest that highly cited papers are robustly encoded in model representations, informing evaluation and design of bibliographic tools and highlighting the need for data-aware mitigation of hallucinations across domains and models.

Abstract

Large language models (LLMs) have been increasingly applied to a wide range of tasks, from natural language understanding to code generation. While they have also been used to assist in bibliographic recommendation, the hallucination of non-existent papers remains a major issue. Building on prior studies, this study hypothesizes that an LLM's ability to correctly produce bibliographic information depends on whether the underlying knowledge is generated or memorized, with highly cited papers (i.e., more frequently appear in the training corpus) showing lower hallucination rates. We therefore assume citation count as a proxy for training data redundancy (i.e., the frequency with which a given bibliographic record is repeatedly represented in the pretraining corpus) and investigate how citation frequency affects hallucinated references in LLM outputs. Using GPT-4.1, we generated and manually verified 100 bibliographic records across twenty computer-science domains, and measured factual consistency via cosine similarity between generated and authentic metadata. The results revealed that (i) hallucination rates vary across research domains, (ii) citation count is strongly correlated with factual accuracy, and (iii) bibliographic information becomes almost verbatimly memorized beyond approximately 1,000 citations. These findings suggest that highly cited papers are nearly verbatimly retained in the model, indicating a threshold where generalization shifts into memorization.

Hallucinations in Bibliographic Recommendation: Citation Frequency as a Proxy for Training Data Redundancy

TL;DR

This study investigates whether LLM hallucinations in bibliographic recommendations arise from training-data redundancy by using citation counts as a proxy. The authors generate 100 bibliographic records across 20 CS domains with GPT-4.1, validate them via Google Scholar, and measure factual fidelity with cosine similarity on metadata; they reveal domain-dependent hallucination rates and a strong positive link between citation frequency and factual accuracy. A non-linear memorization pattern emerges, with a threshold around citations where highly cited papers are recalled verbatim, implying memorization rather than generalized generation. These findings suggest that highly cited papers are robustly encoded in model representations, informing evaluation and design of bibliographic tools and highlighting the need for data-aware mitigation of hallucinations across domains and models.

Abstract

Large language models (LLMs) have been increasingly applied to a wide range of tasks, from natural language understanding to code generation. While they have also been used to assist in bibliographic recommendation, the hallucination of non-existent papers remains a major issue. Building on prior studies, this study hypothesizes that an LLM's ability to correctly produce bibliographic information depends on whether the underlying knowledge is generated or memorized, with highly cited papers (i.e., more frequently appear in the training corpus) showing lower hallucination rates. We therefore assume citation count as a proxy for training data redundancy (i.e., the frequency with which a given bibliographic record is repeatedly represented in the pretraining corpus) and investigate how citation frequency affects hallucinated references in LLM outputs. Using GPT-4.1, we generated and manually verified 100 bibliographic records across twenty computer-science domains, and measured factual consistency via cosine similarity between generated and authentic metadata. The results revealed that (i) hallucination rates vary across research domains, (ii) citation count is strongly correlated with factual accuracy, and (iii) bibliographic information becomes almost verbatimly memorized beyond approximately 1,000 citations. These findings suggest that highly cited papers are nearly verbatimly retained in the model, indicating a threshold where generalization shifts into memorization.

Paper Structure

This paper contains 11 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Study
  3. Experiments
  4. Methodology
  5. Results
  6. Experiment 1.
  7. Experiment 3.
  8. Experiment 4.
  9. Conclusion
  10. Model Settings
  11. Generated Records

Figures (3)

  • Figure 1: Average factual score by domain. Error bars indicate the 95% confidence interval. Accuracy varies by domain familiarity.
  • Figure 2: Relationship between citation frequency and generation fidelity. Each dot represents a factual citation generated by the model, colored by research domain. The dashed line indicates the fitted linear regression (95% CI in gray). The correlation ($r = 0.75$, $p < .001$) demonstrates a strong log-linear relationship between citations and factual accuracy, with saturation near $\log$(citation) $\approx$ 7.
  • Figure 3: Prompt to generate bibliographic information