Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection

Shiping Yang, Renliang Sun, Xiaojun Wan

TL;DR

This work tackles passage-level hallucination detection in LLM outputs by introducing the PHD benchmark and a zero-resource method, Reverse Validation (RV). RV treats LLMs as implicit knowledge bases and uses a three-stage, query-based process to detect discrepancies without external data, with two prompt-driven variants (Question Generation and Entity Matching). Across PHD and WikiBio-GPT3, RV variants outperform baselines, offering higher precision/recall and lower token costs, though challenges remain for high-knowledge-domain passages and outdated-information cases. The study provides a high-quality dataset, a robust, model-agnostic detection approach, and insights into the limitations of zero-resource methods, informing practical deployment of trustworthy LLM systems.

Abstract

Large Language Models (LLMs) have shown their ability to collaborate effectively with humans in real-world scenarios. However, LLMs are apt to generate hallucinations, i.e., makeup incorrect text and unverified information, which can cause significant damage when deployed for mission-critical tasks. In this paper, we propose a self-check approach based on reverse validation to detect factual errors automatically in a zero-resource fashion. To facilitate future studies and assess different methods, we construct a hallucination detection benchmark named PHD, which is generated by ChatGPT and annotated by human annotators. Contrasting previous studies of zero-resource hallucination detection, our method and benchmark concentrate on passage-level detection instead of sentence-level. We empirically evaluate our method and existing zero-resource detection methods on two datasets. The experimental results demonstrate that the proposed method considerably outperforms the baselines while costing fewer tokens and less time. Furthermore, we manually analyze some hallucination cases that LLM failed to capture, revealing the shared limitation of zero-resource methods.

A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection

TL;DR

This work tackles passage-level hallucination detection in LLM outputs by introducing the PHD benchmark and a zero-resource method, Reverse Validation (RV). RV treats LLMs as implicit knowledge bases and uses a three-stage, query-based process to detect discrepancies without external data, with two prompt-driven variants (Question Generation and Entity Matching). Across PHD and WikiBio-GPT3, RV variants outperform baselines, offering higher precision/recall and lower token costs, though challenges remain for high-knowledge-domain passages and outdated-information cases. The study provides a high-quality dataset, a robust, model-agnostic detection approach, and insights into the limitations of zero-resource methods, informing practical deployment of trustworthy LLM systems.

Abstract

Large Language Models (LLMs) have shown their ability to collaborate effectively with humans in real-world scenarios. However, LLMs are apt to generate hallucinations, i.e., makeup incorrect text and unverified information, which can cause significant damage when deployed for mission-critical tasks. In this paper, we propose a self-check approach based on reverse validation to detect factual errors automatically in a zero-resource fashion. To facilitate future studies and assess different methods, we construct a hallucination detection benchmark named PHD, which is generated by ChatGPT and annotated by human annotators. Contrasting previous studies of zero-resource hallucination detection, our method and benchmark concentrate on passage-level detection instead of sentence-level. We empirically evaluate our method and existing zero-resource detection methods on two datasets. The experimental results demonstrate that the proposed method considerably outperforms the baselines while costing fewer tokens and less time. Furthermore, we manually analyze some hallucination cases that LLM failed to capture, revealing the shared limitation of zero-resource methods.
Paper Structure (33 sections, 1 figure, 8 tables)

This paper contains 33 sections, 1 figure, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Hallucinations of LLMs
  4. Mitigate Hallucinations
  5. The PHD Benchmark
  6. Data Generation
  7. Human Annotation
  8. Labels Definition
  9. Worker Requirements
  10. Qualification Test
  11. Two-stage Annotation
  12. Quality Control
  13. Statistics
  14. Method
  15. Reverse Validation Method
  16. ...and 18 more sections

Figures (1)

  • Figure 1: The overview process of the reverse validation (RV) method