Hallucination Detection and Evaluation of Large Language Model

TL;DR

This work addresses the challenge of hallucinations in large language models by proposing an efficient evaluation framework that integrates the Hughes Hallucination Evaluation Model (HHEM) with a decomposition-based retrieval pipeline. It combines query decomposition, knowledge retrieval/optimization, and HHEM-driven factual verification to detect hallucinations across QA and summarization tasks, achieving substantial speedups over KnowHalu and competitive accuracy, especially when augmented with non-fabrication checks. The study reveals task-dependent performance, with QA benefiting from HHEM’s efficiency while summarization requires segment-based verification to detect localized errors, and it highlights that larger models generally show fewer hallucinations though variability persists in intermediate sizes. The proposed framework emphasizes a balance between computational efficiency and robust factual validation, offering practical guidance for scalable evaluation of LLM-generated content and suggesting directions such as RL-based refinements and enhanced segmentation for future work.

Abstract

Hallucinations in Large Language Models (LLMs) pose a significant challenge, generating misleading or unverifiable content that undermines trust and reliability. Existing evaluation methods, such as KnowHalu, employ multi-stage verification but suffer from high computational costs. To address this, we integrate the Hughes Hallucination Evaluation Model (HHEM), a lightweight classification-based framework that operates independently of LLM-based judgments, significantly improving efficiency while maintaining high detection accuracy. We conduct a comparative analysis of hallucination detection methods across various LLMs, evaluating True Positive Rate (TPR), True Negative Rate (TNR), and Accuracy on question-answering (QA) and summarization tasks. Our results show that HHEM reduces evaluation time from 8 hours to 10 minutes, while HHEM with non-fabrication checking achieves the highest accuracy \(82.2\%\) and TPR \(78.9\%\). However, HHEM struggles with localized hallucinations in summarization tasks. To address this, we introduce segment-based retrieval, improving detection by verifying smaller text components. Additionally, our cumulative distribution function (CDF) analysis indicates that larger models (7B-9B parameters) generally exhibit fewer hallucinations, while intermediate-sized models show higher instability. These findings highlight the need for structured evaluation frameworks that balance computational efficiency with robust factual validation, enhancing the reliability of LLM-generated content.

Figures (5)

• Figure 1: Our work is structured around a comprehensive pipeline designed to identify and rectify hallucinations through a multi-stage factual checking process. The hallucinations are generated by the language model based on the given prompt, which consists of an input source and an instruction. The hallucination detector evaluates the generated response by querying external knowledge and applying the HHEM method to compute a hallucination score. If the score falls below a certain threshold, the response is classified as a hallucination.
• Figure 2: Results of QA dataset-Starling-LM-7B-alpha
• Figure 3: Results of Summarization dataset-Starling-LM-7B-alpha
• Figure 4: Box Plot of Generated-Summarization Word Counts for Different Models
• Figure 5: Cumulative Distribution Function (CDF) of HHEM Scores for Different Language Models