Bolster Hallucination Detection via Prompt-Guided Data Augmentation

TL;DR

This paper tackles the scarcity of labeled data for hallucination detection in LLM outputs. It proposes PALE, which uses prompt-guided data augmentation to generate both truthful and hallucinated QA pairs without extra human annotation, and introduces the Contrastive Mahalanobis Score to assess truthfulness by modeling latent activation distributions as Gaussians. The approach yields state-of-the-art AUROC across multiple QA benchmarks and scales well to larger LLMs, with robust cross-dataset transfer capabilities. The work highlights the practical potential of augmented data and distribution-aware latent-space scoring for reliable hallucination detection in real-world applications.

Abstract

Large language models (LLMs) have garnered significant interest in AI community. Despite their impressive generation capabilities, they have been found to produce misleading or fabricated information, a phenomenon known as hallucinations. Consequently, hallucination detection has become critical to ensure the reliability of LLM-generated content. One primary challenge in hallucination detection is the scarcity of well-labeled datasets containing both truthful and hallucinated outputs. To address this issue, we introduce Prompt-guided data Augmented haLlucination dEtection (PALE), a novel framework that leverages prompt-guided responses from LLMs as data augmentation for hallucination detection. This strategy can generate both truthful and hallucinated data under prompt guidance at a relatively low cost. To more effectively evaluate the truthfulness of the sparse intermediate embeddings produced by LLMs, we introduce an estimation metric called the Contrastive Mahalanobis Score (CM Score). This score is based on modeling the distributions of truthful and hallucinated data in the activation space. CM Score employs a matrix decomposition approach to more accurately capture the underlying structure of these distributions. Importantly, our framework does not require additional human annotations, offering strong generalizability and practicality for real-world applications. Extensive experiments demonstrate that PALE achieves superior hallucination detection performance, outperforming the competitive baseline by a significant margin of 6.55%.

Figures (7)

• Figure 1: The pipeline of our proposed PALE. Our PALE comprises three steps: 1) Utilizing state‑of‑the‑art LLMs to generate both truthful and hallucinated data; 2) Employing the Contrastive Mahalanobis Score to determine the truthiness of test samples.
• Figure 2: Prompt engineering technology illustration for data augmentation on state-of-the-art LLMs.
• Figure 3: (a) Generalization across four datasets. Rows indicate the source dataset, while columns indicate the target dataset. (b) Impact of different layers. All values represent AUROC scores based on LLaMA‑3.1‑7B.
• Figure 4: (a) Ablation study on the different LLMs setting for data augmentation. (b) Comparison of direct classifier based detection versus CM Score based detection. All results are reported as AUROC using LLaMA‑3.1‑7B.
• Figure 5: Score distribution visualization for HaloScope vs. our method.

Theorems & Definitions (4)

• Definition 1: LLM generation
• Definition 2: Hallucination detection
• Definition 3: Prompt-guided data Generation
• Definition 4: Empirical dataset