Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Poly-FEVER: A Multilingual Fact Verification Benchmark for Hallucination Detection in Large Language Models

Hanzhi Zhang, Sumera Anjum, Heng Fan, Weijian Zheng, Yan Huang, Yunhe Feng

TL;DR

Poly-FEVER presents a large-scale multilingual benchmark for hallucination detection in LLMs by extending FEVER-derived resources to 11 languages, totaling 77,973 claims. It analyzes models such as ChatGPT and LLaMA under language-specific and classification prompts, while employing LDA topic modeling and web-based bias analyses to understand cross-lingual patterns. The study finds that topic distribution and web data availability influence hallucination rates, and shows that topic-aware and retrieval-augmented approaches can help lower-resource languages though not uniformly. The benchmark enables cross-lingual evaluation and highlights the need for language-adaptive methods to improve multilingual factuality and reliability in AI systems.

Abstract

Hallucinations in generative AI, particularly in Large Language Models (LLMs), pose a significant challenge to the reliability of multilingual applications. Existing benchmarks for hallucination detection focus primarily on English and a few widely spoken languages, lacking the breadth to assess inconsistencies in model performance across diverse linguistic contexts. To address this gap, we introduce Poly-FEVER, a large-scale multilingual fact verification benchmark specifically designed for evaluating hallucination detection in LLMs. Poly-FEVER comprises 77,973 labeled factual claims spanning 11 languages, sourced from FEVER, Climate-FEVER, and SciFact. It provides the first large-scale dataset tailored for analyzing hallucination patterns across languages, enabling systematic evaluation of LLMs such as ChatGPT and the LLaMA series. Our analysis reveals how topic distribution and web resource availability influence hallucination frequency, uncovering language-specific biases that impact model accuracy. By offering a multilingual benchmark for fact verification, Poly-FEVER facilitates cross-linguistic comparisons of hallucination detection and contributes to the development of more reliable, language-inclusive AI systems. The dataset is publicly available to advance research in responsible AI, fact-checking methodologies, and multilingual NLP, promoting greater transparency and robustness in LLM performance. The proposed Poly-FEVER is available at: https://huggingface.co/datasets/HanzhiZhang/Poly-FEVER.

Poly-FEVER: A Multilingual Fact Verification Benchmark for Hallucination Detection in Large Language Models

TL;DR

Poly-FEVER presents a large-scale multilingual benchmark for hallucination detection in LLMs by extending FEVER-derived resources to 11 languages, totaling 77,973 claims. It analyzes models such as ChatGPT and LLaMA under language-specific and classification prompts, while employing LDA topic modeling and web-based bias analyses to understand cross-lingual patterns. The study finds that topic distribution and web data availability influence hallucination rates, and shows that topic-aware and retrieval-augmented approaches can help lower-resource languages though not uniformly. The benchmark enables cross-lingual evaluation and highlights the need for language-adaptive methods to improve multilingual factuality and reliability in AI systems.

Abstract

Hallucinations in generative AI, particularly in Large Language Models (LLMs), pose a significant challenge to the reliability of multilingual applications. Existing benchmarks for hallucination detection focus primarily on English and a few widely spoken languages, lacking the breadth to assess inconsistencies in model performance across diverse linguistic contexts. To address this gap, we introduce Poly-FEVER, a large-scale multilingual fact verification benchmark specifically designed for evaluating hallucination detection in LLMs. Poly-FEVER comprises 77,973 labeled factual claims spanning 11 languages, sourced from FEVER, Climate-FEVER, and SciFact. It provides the first large-scale dataset tailored for analyzing hallucination patterns across languages, enabling systematic evaluation of LLMs such as ChatGPT and the LLaMA series. Our analysis reveals how topic distribution and web resource availability influence hallucination frequency, uncovering language-specific biases that impact model accuracy. By offering a multilingual benchmark for fact verification, Poly-FEVER facilitates cross-linguistic comparisons of hallucination detection and contributes to the development of more reliable, language-inclusive AI systems. The dataset is publicly available to advance research in responsible AI, fact-checking methodologies, and multilingual NLP, promoting greater transparency and robustness in LLM performance. The proposed Poly-FEVER is available at: https://huggingface.co/datasets/HanzhiZhang/Poly-FEVER.

Paper Structure

This paper contains 25 sections, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Poly-FEVER Benchmark
  4. Poly-FEVER Overview
  5. Structure of Poly-FEVER
  6. LDA on Topic Distribution
  7. Language Selection
  8. Multilingual Claim Translation
  9. Hallucination Causes Exploration
  10. Model Performance
  11. Effect of Prompt Design
  12. Self-Detection of Hallucinations in LLMs
  13. Web Search on References Bias
  14. Evaluating External Knowledge for Hallucination Reduction
  15. Evaluation and Analytics
  16. ...and 10 more sections

Figures (3)

  • Figure 1: Web search distribution on multilingual claims. Middle 50% of search counts inside each box, mean values for every language are connected.
  • Figure 2: Detection accuracy on web search count. Bubble sizes depend on data variances to mean values on web search counts of each language.
  • Figure 3: Average accuracy (represented by color) across topics by language, with standard deviation indicated by annotation. The color intensity reflects accuracy levels, where higher accuracy is represented by stronger shades, while lower accuracy is indicated by lighter tones. Standard deviation values illustrate the degree of variability within each topic, highlighting cases where the model exhibits inconsistent fact-checking performance across languages. The bottom row aggregates the mean accuracy and standard deviation for each topic across all languages, providing an overall assessment of topic difficulty and model robustness.