Quantifying Genuine Awareness in Hallucination Prediction Beyond Question-Side Shortcuts

TL;DR

This work proposes a methodology for measuring this effect without requiring human labor, Approximate Question-side Effect (AQE), and reveals that existing hallucination detection methods rely heavily on benchmark hacking.

Abstract

Many works have proposed methodologies for language model (LM) hallucination detection and reported seemingly strong performance. However, we argue that the reported performance to date reflects not only a model's genuine awareness of its internal information, but also awareness derived purely from question-side information (e.g., benchmark hacking). While benchmark hacking can be effective for boosting hallucination detection score on existing benchmarks, it does not generalize to out-of-domain settings and practical usage. Nevertheless, disentangling how much of a model's hallucination detection performance arises from question-side awareness is non-trivial. To address this, we propose a methodology for measuring this effect without requiring human labor, Approximate Question-side Effect (AQE). Our analysis using AQE reveals that existing hallucination detection methods rely heavily on benchmark hacking.

Figures (8)

• Figure 1: Pipeline for the prediction of knowing (prediction of hallucination).
• Figure 2: The portion of $k=True$ for each domain, by LLaMA-3-8B model on the ParaRel train set. The rate is skewed toward 0 or 1 by domain, rather than being centered around 0.5.
• Figure 3: Structural analogy between 1) dense retriever and 2) causal LM.
• Figure 4: Y-axis is the top-7 candidates of the first token of the answer to the question "Please give me an explanation about Breaking Dawn". The X-axis is the probability for each candidate. Left is for one-word prompt, and the Right is for normal prompt.
• Figure 5: Probability pattern of the hallucinated answer, by LLaMA3-8B. Each bar stands for the probability (0,1) of the corresponding token.