Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Not Wrong, But Untrue: LLM Overconfidence in Document-Based Queries

Nick Hagar, Wilma Agustianto, Nicholas Diakopoulos

TL;DR

The paper investigates LLM hallucinations in document-based journalism tasks by evaluating ChatGPT, Gemini, and NotebookLM on a 300-document TikTok-related corpus, varying prompt specificity and context size. Using sentence-level annotation with the Rawte taxonomy, it finds a 30% overall hallucination rate, with Gemini and ChatGPT around 40% and NotebookLM at 13%, and identifies interpretive overconfidence as the dominant failure mode. The authors argue that current hallucination taxonomies miss journalism-specific errors and propose extensions focused on attribution and sourcing, advocating for newsroom-oriented architectures that enforce provenance over fluency. The work highlights a fundamental epistemological mismatch between journalistic practice and LLMs, with practical implications for tool selection, verification workflows, and the design of journalist-friendly AI systems that prioritize explicit citations and traceable argumentation.

Abstract

Large language models (LLMs) are increasingly used in newsroom workflows, but their tendency to hallucinate poses risks to core journalistic practices of sourcing, attribution, and accuracy. We evaluate three widely used tools - ChatGPT, Gemini, and NotebookLM - on a reporting-style task grounded in a 300-document corpus related to TikTok litigation and policy in the U.S. We vary prompt specificity and context size and annotate sentence-level outputs using a taxonomy to measure hallucination type and severity. Across our sample, 30% of model outputs contained at least one hallucination, with rates approximately three times higher for Gemini and ChatGPT (40%) than for NotebookLM (13%). Qualitatively, most errors did not involve invented entities or numbers; instead, we observed interpretive overconfidence - models added unsupported characterizations of sources and transformed attributed opinions into general statements. These patterns reveal a fundamental epistemological mismatch: While journalism requires explicit sourcing for every claim, LLMs generate authoritative-sounding text regardless of evidentiary support. We propose journalism-specific extensions to existing hallucination taxonomies and argue that effective newsroom tools need architectures that enforce accurate attribution rather than optimize for fluency.

Not Wrong, But Untrue: LLM Overconfidence in Document-Based Queries

TL;DR

The paper investigates LLM hallucinations in document-based journalism tasks by evaluating ChatGPT, Gemini, and NotebookLM on a 300-document TikTok-related corpus, varying prompt specificity and context size. Using sentence-level annotation with the Rawte taxonomy, it finds a 30% overall hallucination rate, with Gemini and ChatGPT around 40% and NotebookLM at 13%, and identifies interpretive overconfidence as the dominant failure mode. The authors argue that current hallucination taxonomies miss journalism-specific errors and propose extensions focused on attribution and sourcing, advocating for newsroom-oriented architectures that enforce provenance over fluency. The work highlights a fundamental epistemological mismatch between journalistic practice and LLMs, with practical implications for tool selection, verification workflows, and the design of journalist-friendly AI systems that prioritize explicit citations and traceable argumentation.

Abstract

Large language models (LLMs) are increasingly used in newsroom workflows, but their tendency to hallucinate poses risks to core journalistic practices of sourcing, attribution, and accuracy. We evaluate three widely used tools - ChatGPT, Gemini, and NotebookLM - on a reporting-style task grounded in a 300-document corpus related to TikTok litigation and policy in the U.S. We vary prompt specificity and context size and annotate sentence-level outputs using a taxonomy to measure hallucination type and severity. Across our sample, 30% of model outputs contained at least one hallucination, with rates approximately three times higher for Gemini and ChatGPT (40%) than for NotebookLM (13%). Qualitatively, most errors did not involve invented entities or numbers; instead, we observed interpretive overconfidence - models added unsupported characterizations of sources and transformed attributed opinions into general statements. These patterns reveal a fundamental epistemological mismatch: While journalism requires explicit sourcing for every claim, LLMs generate authoritative-sounding text regardless of evidentiary support. We propose journalism-specific extensions to existing hallucination taxonomies and argue that effective newsroom tools need architectures that enforce accurate attribution rather than optimize for fluency.

Paper Structure

This paper contains 15 sections, 1 figure.

Table of Contents

  1. Introduction
  2. Background
  3. Data
  4. Methods
  5. Query Approach
  6. LLM Tools Used
  7. Response Annotation and Evaluation
  8. Results
  9. Hallucination Prevalence
  10. Qualitative Characteristics of Observed Hallucinations
  11. Discussion
  12. Limitations
  13. Practical Implications
  14. Conclusion
  15. LLM Queries

Figures (1)

  • Figure 1: Overall rate of hallucination per tool. Gemini and ChatGPT return the most responses with hallucinations.