Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hallucinate or Memorize? The Two Sides of Probabilistic Learning in Large Language Models

Junichiro Niimi

TL;DR

This study investigates why LLMs fabricate bibliographic references and demonstrates that training data redundancy, proxied by citation frequency, governs whether a citation is memorized or generated. Using GPT-4.1, the authors generated 100 bibliographic records across 20 computer science topics and evaluated them with manual verification and semantic similarity via Sentence-BERT. They reveal a strong positive relation between citation frequency and factual fidelity, with memorization becoming nearly deterministic beyond about 1,248 citations and two memorization thresholds near 90 and 1,248 citations. The findings support a unified view that hallucination and memorization are two sides of the same probabilistic learning process, with practical implications for verifying LLM-generated citations and guiding retrieval-augmented strategies to mitigate non-existent references.

Abstract

Large language models (LLMs) have been increasingly applied to a wide range of tasks, from natural language understanding to code generation. While they have also been used to assist in citation recommendation, the hallucination of non-existent papers remains a major issue. Building on prior studies, this study hypothesizes that an LLM's ability to correctly produce bibliographic records depends on whether the underlying knowledge is generated or memorized, with highly cited papers (i.e., more frequently appear in the pretraining corpus) showing lower hallucination rates. We therefore assume citation count as a proxy for training data redundancy (i.e., the frequency with which a given bibliographic record appears in the pretraining corpus) and investigate how citation frequency affects hallucinated references in LLM outputs. Using GPT-4.1, we generated and manually verified 100 citations across twenty computer-science domains, and measured factual consistency via cosine similarity between generated and authentic metadata. The results revealed that (i) citation count is strongly correlated with factual accuracy, (ii) bibliographic information becomes almost verbatim memorized beyond roughly 1,000 citations, and (iii) memory interference occurs when multiple highly cited papers share similar content. These findings indicate a threshold where generalization shifts into memorization, with highly cited papers being nearly verbatim retained in the model.

Hallucinate or Memorize? The Two Sides of Probabilistic Learning in Large Language Models

TL;DR

This study investigates why LLMs fabricate bibliographic references and demonstrates that training data redundancy, proxied by citation frequency, governs whether a citation is memorized or generated. Using GPT-4.1, the authors generated 100 bibliographic records across 20 computer science topics and evaluated them with manual verification and semantic similarity via Sentence-BERT. They reveal a strong positive relation between citation frequency and factual fidelity, with memorization becoming nearly deterministic beyond about 1,248 citations and two memorization thresholds near 90 and 1,248 citations. The findings support a unified view that hallucination and memorization are two sides of the same probabilistic learning process, with practical implications for verifying LLM-generated citations and guiding retrieval-augmented strategies to mitigate non-existent references.

Abstract

Large language models (LLMs) have been increasingly applied to a wide range of tasks, from natural language understanding to code generation. While they have also been used to assist in citation recommendation, the hallucination of non-existent papers remains a major issue. Building on prior studies, this study hypothesizes that an LLM's ability to correctly produce bibliographic records depends on whether the underlying knowledge is generated or memorized, with highly cited papers (i.e., more frequently appear in the pretraining corpus) showing lower hallucination rates. We therefore assume citation count as a proxy for training data redundancy (i.e., the frequency with which a given bibliographic record appears in the pretraining corpus) and investigate how citation frequency affects hallucinated references in LLM outputs. Using GPT-4.1, we generated and manually verified 100 citations across twenty computer-science domains, and measured factual consistency via cosine similarity between generated and authentic metadata. The results revealed that (i) citation count is strongly correlated with factual accuracy, (ii) bibliographic information becomes almost verbatim memorized beyond roughly 1,000 citations, and (iii) memory interference occurs when multiple highly cited papers share similar content. These findings indicate a threshold where generalization shifts into memorization, with highly cited papers being nearly verbatim retained in the model.

Paper Structure

This paper contains 19 sections, 2 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED STUDY
  3. Citation recommendation
  4. Hallucination in LLMs
  5. EXPERIMENTAL DESIGN
  6. Model settings
  7. Sample size
  8. Data source
  9. Human evaluation
  10. Computational evaluation
  11. RESULTS AND DISCUSSIONS
  12. Experiment 1: Citation frequency and factual accuracy
  13. Experiment 2: Relationship between citations and fidelity
  14. Experiment 3: Identifying memorization thresholds
  15. Error Analysis
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Prompt to generate bibliographic information
  • Figure 2: Relationship between citation frequency and generation fidelity. Each dot represents a factual record ($score > 0$), colored by research domain. The regression line indicates fitted linear regression with 95% confidence interval (gray band). Strong correlation ($r = 0.75$, $p < .001$) demonstrates a log-linear scaling relationship. Note the saturation near $\log(\text{citation}) \approx 7$, suggesting a memorization threshold.