Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models

TL;DR

This work tackles the pervasive problem of hallucinations in large language models by introducing RelD, a robust discriminator trained on a large bilingual QA dataset RelQA to detect unreliable model outputs. RelD is built by reframing supervision from a regression objective to a 10-class then binary classification using a weighted-average probability scheme and ELECTRA as backbone, with inputs comprising questions, contexts, and LLM-generated answers. RelQA aggregates nine QA datasets, applies personalized prompts and long-context handling, and is evaluated with a comprehensive metric suite (LLM-assessment, human, machine, composite) to label answer reliability. Experiments show RelD achieves strong automatic and human-aligned performance across several LLMs and generalizes well to in-distribution and out-of-distribution data, providing detailed analyses of hallucination types and informing future mitigation strategies.

Abstract

Large Language Models (LLMs) have gained widespread adoption in various natural language processing tasks, including question answering and dialogue systems. However, a major drawback of LLMs is the issue of hallucination, where they generate unfaithful or inconsistent content that deviates from the input source, leading to severe consequences. In this paper, we propose a robust discriminator named RelD to effectively detect hallucination in LLMs' generated answers. RelD is trained on the constructed RelQA, a bilingual question-answering dialogue dataset along with answers generated by LLMs and a comprehensive set of metrics. Our experimental results demonstrate that the proposed RelD successfully detects hallucination in the answers generated by diverse LLMs. Moreover, it performs well in distinguishing hallucination in LLMs' generated answers from both in-distribution and out-of-distribution datasets. Additionally, we also conduct a thorough analysis of the types of hallucinations that occur and present valuable insights. This research significantly contributes to the detection of reliable answers generated by LLMs and holds noteworthy implications for mitigating hallucination in the future work.

Figures (11)

• Figure 1: The given answer, produced by ChatGPT, exhibits "Hallucinations" by incorrectly treating "Shuren Zhou" and "Xun Lu" as separate individuals, despite they referring to the same person.
• Figure 2: A data exploratory analysis of the constructed RelQA based on different metrics.
• Figure 3: The process of building the discriminator RelD, which is trained on the constructed dataset RelQA and used to detect hallucination of LLMs' generated answers.
• Figure 4: The visualization of RelD's performance among the selected LLMs on the validation dataset, including both automatic and human-in-the-loop metrics.
• Figure 5: Performance of RelD on automatic metrics (a) and human-in-the-loop metrics (b)(c), including results on IID validation dataset (b) and OOD dataset (c) among the selected LLMs.