Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Language Models Hallucinate, but May Excel at Fact Verification

Jian Guan, Jesse Dodge, David Wadden, Minlie Huang, Hao Peng

TL;DR

Surprisingly, FLAN-T5-11B, the least factual generator in this study, performs the best as a fact verifier, even outperforming more capable LLMs like GPT3.5 and ChatGPT.

Abstract

Recent progress in natural language processing (NLP) owes much to remarkable advances in large language models (LLMs). Nevertheless, LLMs frequently "hallucinate," resulting in non-factual outputs. Our carefully-designed human evaluation substantiates the serious hallucination issue, revealing that even GPT-3.5 produces factual outputs less than 25% of the time. This underscores the importance of fact verifiers in order to measure and incentivize progress. Our systematic investigation affirms that LLMs can be repurposed as effective fact verifiers with strong correlations with human judgments. Surprisingly, FLAN-T5-11B, the least factual generator in our study, performs the best as a fact verifier, even outperforming more capable LLMs like GPT3.5 and ChatGPT. Delving deeper, we analyze the reliance of these LLMs on high-quality evidence, as well as their deficiencies in robustness and generalization ability. Our study presents insights for developing trustworthy generation models.

Language Models Hallucinate, but May Excel at Fact Verification

TL;DR

Surprisingly, FLAN-T5-11B, the least factual generator in this study, performs the best as a fact verifier, even outperforming more capable LLMs like GPT3.5 and ChatGPT.

Abstract

Recent progress in natural language processing (NLP) owes much to remarkable advances in large language models (LLMs). Nevertheless, LLMs frequently "hallucinate," resulting in non-factual outputs. Our carefully-designed human evaluation substantiates the serious hallucination issue, revealing that even GPT-3.5 produces factual outputs less than 25% of the time. This underscores the importance of fact verifiers in order to measure and incentivize progress. Our systematic investigation affirms that LLMs can be repurposed as effective fact verifiers with strong correlations with human judgments. Surprisingly, FLAN-T5-11B, the least factual generator in our study, performs the best as a fact verifier, even outperforming more capable LLMs like GPT3.5 and ChatGPT. Delving deeper, we analyze the reliance of these LLMs on high-quality evidence, as well as their deficiencies in robustness and generalization ability. Our study presents insights for developing trustworthy generation models.
Paper Structure (50 sections, 2 equations, 5 figures, 29 tables)

This paper contains 50 sections, 2 equations, 5 figures, 29 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Hallucination
  4. Fact Verification
  5. LLMs as Evaluators
  6. Quantifying LLMs' Hallucination
  7. Generation Models
  8. Generation Tasks
  9. Human Annotation
  10. Repurposing LLMs as Fact Verifiers
  11. Evaluation Sets
  12. Verification Method
  13. Compared Verifiers
  14. Results
  15. Results on wks&dss
  16. ...and 35 more sections

Figures (5)

  • Figure 1: The annotation interface for ParaGen; that for SentCom is similar.
  • Figure 2: Distribution of different labels arcross sentence positions in ParaGen (Para).
  • Figure 3: Pearson's correlations when verifying outputs from different generation models. Top: SentCom; Bottom: ParaGen (Sent). We calculate the standard deviation (black bars) across four instructions in §\ref{['robust']}.
  • Figure 4: Precision and recall on the unfactual category varying with entity types. "Non-NE" means non-named-entity common words (e.g., "computer").
  • Figure 5: Precision and recall scores with retrieved evidence in wks (Left: Factual Right: Unfactual). Takeaway: ChatGPT prefers to predict factual.