Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Visualizing and Benchmarking LLM Factual Hallucination Tendencies via Internal State Analysis and Clustering

Nathan Mao, Varun Kaushik, Shreya Shivkumar, Parham Sharafoleslami, Kevin Zhu, Sunishchal Dev

TL;DR

The paper tackles LLM factual hallucinations by introducing FalseCite, a benchmark that pairs false claims with fabricated citations to study citation-driven and content-based hallucinations. It demonstrates that deceptive citations significantly elevate hallucination rates across multiple models, with random citations often having the strongest effect and larger relative impacts on the more capable model GPT-4o-mini. Beyond evaluation, it analyzes internal model dynamics by capturing and clustering hidden-state vectors, uncovering a horn-shaped trajectory in the representation space and identifying layers more associated with hallucination. The work provides a foundation for benchmarking and mitigating hallucinations in future LLM research through internal-state visualization and targeted dataset design.

Abstract

Large Language Models (LLMs) often hallucinate, generating nonsensical or false information that can be especially harmful in sensitive fields such as medicine or law. To study this phenomenon systematically, we introduce FalseCite, a curated dataset designed to capture and benchmark hallucinated responses induced by misleading or fabricated citations. Running GPT-4o-mini, Falcon-7B, and Mistral 7-B through FalseCite, we observed a noticeable increase in hallucination activity for false claims with deceptive citations, especially in GPT-4o-mini. Using the responses from FalseCite, we can also analyze the internal states of hallucinating models, visualizing and clustering the hidden state vectors. From this analysis, we noticed that the hidden state vectors, regardless of hallucination or non-hallucination, tend to trace out a distinct horn-like shape. Our work underscores FalseCite's potential as a foundation for evaluating and mitigating hallucinations in future LLM research.

Visualizing and Benchmarking LLM Factual Hallucination Tendencies via Internal State Analysis and Clustering

TL;DR

The paper tackles LLM factual hallucinations by introducing FalseCite, a benchmark that pairs false claims with fabricated citations to study citation-driven and content-based hallucinations. It demonstrates that deceptive citations significantly elevate hallucination rates across multiple models, with random citations often having the strongest effect and larger relative impacts on the more capable model GPT-4o-mini. Beyond evaluation, it analyzes internal model dynamics by capturing and clustering hidden-state vectors, uncovering a horn-shaped trajectory in the representation space and identifying layers more associated with hallucination. The work provides a foundation for benchmarking and mitigating hallucinations in future LLM research through internal-state visualization and targeted dataset design.

Abstract

Large Language Models (LLMs) often hallucinate, generating nonsensical or false information that can be especially harmful in sensitive fields such as medicine or law. To study this phenomenon systematically, we introduce FalseCite, a curated dataset designed to capture and benchmark hallucinated responses induced by misleading or fabricated citations. Running GPT-4o-mini, Falcon-7B, and Mistral 7-B through FalseCite, we observed a noticeable increase in hallucination activity for false claims with deceptive citations, especially in GPT-4o-mini. Using the responses from FalseCite, we can also analyze the internal states of hallucinating models, visualizing and clustering the hidden state vectors. From this analysis, we noticed that the hidden state vectors, regardless of hallucination or non-hallucination, tend to trace out a distinct horn-like shape. Our work underscores FalseCite's potential as a foundation for evaluating and mitigating hallucinations in future LLM research.
Paper Structure (18 sections, 5 figures, 9 tables)

This paper contains 18 sections, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology of Data Generation
  4. Data Sources
  5. Generation
  6. Results
  7. Results Analysis
  8. Discussion
  9. Activation Capture
  10. Activation Capture Framework
  11. Vector organization and representation
  12. Clustering
  13. Limitations
  14. Extended Results
  15. Citation Templates and Sources
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Graphic of the expert model (GPT-4.1) believing the false citation generated by Mistral-7B because the citation is plausible.
  • Figure 2: Graphic of the expert model (GPT-4.1) marking Mistral-7B's implausible false citation as hallucinated
  • Figure 3: Example of token-level hallucination labeling produced by the expert model. Tokens labeled with 1 correspond to hallucinations, while tokens labeled with 0 correspond to factual content.
  • Figure 4: Illustration of attention vector extraction. The highlighted region corresponds to the attention of the most recently generated token over the input sequence.
  • Figure 5: The clusters of hidden-state vectors on the left, next to the same vectors with layers labeled by color and visualized on the right.