Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Early Detection of Hallucinations in Factual Question Answering

Ben Snyder, Marius Moisescu, Muhammad Bilal Zafar

TL;DR

This work tackles the problem of factual hallucinations in open-ended question answering by leveraging generation artifacts from the input, output, and internal states of Transformer-based models. It proposes four artifact families—Softmax probabilities, Integrated Gradients attributions, self-attention scores, and fully-connected activations—and trains four binary classifiers to detect hallucinations, achieving robust AUROC scores (often above $0.70$, with peaks near $0.80$) and demonstrating that signals can appear before the first generated token. The study systematically analyzes two datasets (T-REx and TriviaQA) and three model families (LAM, OPT, Falcon), showing that internal-state-based detectors, especially self-attention and FC activations, are highly predictive across tasks, while IG-based signals are less reliable. The results suggest that attaching lightweight hallucination detectors to deployed systems could flag potentially incorrect factual content in real time, though the approach requires access to model internals and gradients, and future work should extend to broader datasets and more complex retrieval settings.

Abstract

While large language models (LLMs) have taken great strides towards helping humans with a plethora of tasks, hallucinations remain a major impediment towards gaining user trust. The fluency and coherence of model generations even when hallucinating makes detection a difficult task. In this work, we explore if the artifacts associated with the model generations can provide hints that the generation will contain hallucinations. Specifically, we probe LLMs at 1) the inputs via Integrated Gradients based token attribution, 2) the outputs via the Softmax probabilities, and 3) the internal state via self-attention and fully-connected layer activations for signs of hallucinations on open-ended question answering tasks. Our results show that the distributions of these artifacts tend to differ between hallucinated and non-hallucinated generations. Building on this insight, we train binary classifiers that use these artifacts as input features to classify model generations into hallucinations and non-hallucinations. These hallucination classifiers achieve up to $0.80$ AUROC. We also show that tokens preceding a hallucination can already predict the subsequent hallucination even before it occurs.

On Early Detection of Hallucinations in Factual Question Answering

TL;DR

This work tackles the problem of factual hallucinations in open-ended question answering by leveraging generation artifacts from the input, output, and internal states of Transformer-based models. It proposes four artifact families—Softmax probabilities, Integrated Gradients attributions, self-attention scores, and fully-connected activations—and trains four binary classifiers to detect hallucinations, achieving robust AUROC scores (often above , with peaks near ) and demonstrating that signals can appear before the first generated token. The study systematically analyzes two datasets (T-REx and TriviaQA) and three model families (LAM, OPT, Falcon), showing that internal-state-based detectors, especially self-attention and FC activations, are highly predictive across tasks, while IG-based signals are less reliable. The results suggest that attaching lightweight hallucination detectors to deployed systems could flag potentially incorrect factual content in real time, though the approach requires access to model internals and gradients, and future work should extend to broader datasets and more complex retrieval settings.

Abstract

While large language models (LLMs) have taken great strides towards helping humans with a plethora of tasks, hallucinations remain a major impediment towards gaining user trust. The fluency and coherence of model generations even when hallucinating makes detection a difficult task. In this work, we explore if the artifacts associated with the model generations can provide hints that the generation will contain hallucinations. Specifically, we probe LLMs at 1) the inputs via Integrated Gradients based token attribution, 2) the outputs via the Softmax probabilities, and 3) the internal state via self-attention and fully-connected layer activations for signs of hallucinations on open-ended question answering tasks. Our results show that the distributions of these artifacts tend to differ between hallucinated and non-hallucinated generations. Building on this insight, we train binary classifiers that use these artifacts as input features to classify model generations into hallucinations and non-hallucinations. These hallucination classifiers achieve up to AUROC. We also show that tokens preceding a hallucination can already predict the subsequent hallucination even before it occurs.
Paper Structure (23 sections, 13 figures, 9 tables)

This paper contains 23 sections, 13 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Detecting Hallucinations
  4. Setup
  5. Artifacts for Detecting Hallucinations
  6. Softmax probabilities
  7. Feature attributions
  8. Self-attention and Hidden Activations
  9. Hallucination Detection Classifiers
  10. Experimental Setup
  11. Datasets
  12. T-REx
  13. TriviaQA
  14. Models
  15. Infrastructure and Parameters
  16. ...and 8 more sections

Figures (13)

  • Figure 1: [TriviaQA dataset] The 2D TSNE distribution of the self-attention scores. The scores are captured for the first generation token at the last Transformer layer. The distributions are different between hallucinated v.s. non-hallucinated generations though the differences are more pronounced for some models than the others.
  • Figure 2: [Falcon-40B on TriviaQA dataset] Model artifacts differ between hallucinated (top row) and non-hallucinated outputs (bottom row). Figure \ref{['fig:example_softmax']} shows the Softmax distribution for top-10 tokens at the first generation position. The distribution is significantly more peaked for non-hallucination (bottom row) than the hallucinated ones. Figure \ref{['fig:example_ig']} shows the input feature attribution scores computed using IG method for the same pair of hallucinated (top) and non-hallucinated (bottom) outputs. Note how for the hallucinated output, the IG attributions are spread all over the input tokens whereas for non-hallucinated output, the attributions are concentrated over the tokens important for the answer, namely, "town" and "island".
  • Figure 3: [ FAL-40B on TriviaQA dataset] 2D TSNE projections of the model artifacts for hallucinations and non-hallucinations.
  • Figure 4: [ FAL-40B on TriviaQA dataset] Softmax entropy at different generation steps. The difference in entropy between hallucinated and non-hallucinated outputs does not vary much with a change in the generation steps.
  • Figure 5: [ FAL-40B on TriviaQA dataset] AUROC of the hallucination detectors using self-attention and fully-connected activations at different layers. The performance is better at later layers but has diminishing returns.
  • ...and 8 more figures