Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WildHallucinations: Evaluating Long-form Factuality in LLMs with Real-World Entity Queries

Wenting Zhao, Tanya Goyal, Yu Ying Chiu, Liwei Jiang, Benjamin Newman, Abhilasha Ravichander, Khyathi Chandu, Ronan Le Bras, Claire Cardie, Yuntian Deng, Yejin Choi

TL;DR

WildHallucinations targets the gap in factuality evaluation by using real-world, non-Wikipedia entities extracted from user-chatbot interactions and vetted against a web-sourced knowledge base. The approach couples entity-level prompting with an automatic fact-checking pipeline (FActScore) to produce two factuality metrics, WildFActScore and WildFActScore-Strict, across 7,919 entities and 15 LLMs. Key findings show higher hallucination rates for entities without Wikipedia pages and notable domain-dependent variability; retrieval helps but does not eliminate hallucinations, and open-weight models lag behind closed-weight counterparts. The benchmark provides a scalable, domain-rich, automatic evaluation framework that can guide model development and ongoing benchmarking, with public release on Hugging Face to support broad adoption.

Abstract

While hallucinations of large language models (LLMs) prevail as a major challenge, existing evaluation benchmarks on factuality do not cover the diverse domains of knowledge that the real-world users of LLMs seek information about. To bridge this gap, we introduce WildHallucinations, a benchmark that evaluates factuality. It does so by prompting LLMs to generate information about entities mined from user-chatbot conversations in the wild. These generations are then automatically fact-checked against a systematically curated knowledge source collected from web search. Notably, half of these real-world entities do not have associated Wikipedia pages. We evaluate 118,785 generations from 15 LLMs on 7,919 entities. We find that LLMs consistently hallucinate more on entities without Wikipedia pages and exhibit varying hallucination rates across different domains. Finally, given the same base models, adding a retrieval component only slightly reduces hallucinations but does not eliminate hallucinations.

WildHallucinations: Evaluating Long-form Factuality in LLMs with Real-World Entity Queries

TL;DR

WildHallucinations targets the gap in factuality evaluation by using real-world, non-Wikipedia entities extracted from user-chatbot interactions and vetted against a web-sourced knowledge base. The approach couples entity-level prompting with an automatic fact-checking pipeline (FActScore) to produce two factuality metrics, WildFActScore and WildFActScore-Strict, across 7,919 entities and 15 LLMs. Key findings show higher hallucination rates for entities without Wikipedia pages and notable domain-dependent variability; retrieval helps but does not eliminate hallucinations, and open-weight models lag behind closed-weight counterparts. The benchmark provides a scalable, domain-rich, automatic evaluation framework that can guide model development and ongoing benchmarking, with public release on Hugging Face to support broad adoption.

Abstract

While hallucinations of large language models (LLMs) prevail as a major challenge, existing evaluation benchmarks on factuality do not cover the diverse domains of knowledge that the real-world users of LLMs seek information about. To bridge this gap, we introduce WildHallucinations, a benchmark that evaluates factuality. It does so by prompting LLMs to generate information about entities mined from user-chatbot conversations in the wild. These generations are then automatically fact-checked against a systematically curated knowledge source collected from web search. Notably, half of these real-world entities do not have associated Wikipedia pages. We evaluate 118,785 generations from 15 LLMs on 7,919 entities. We find that LLMs consistently hallucinate more on entities without Wikipedia pages and exhibit varying hallucination rates across different domains. Finally, given the same base models, adding a retrieval component only slightly reduces hallucinations but does not eliminate hallucinations.
Paper Structure (40 sections, 7 figures, 8 tables)

This paper contains 40 sections, 7 figures, 8 tables.

Table of Contents

  1. Introduction
  2. The WildHallucinations Benchmark
  3. Dataset Creation
  4. Entity Extraction.
  5. Building Knowledge Source.
  6. Dataset Statistics
  7. Entity Domains.
  8. Rare Entities.
  9. Evaluation Pipeline
  10. Automatic Fact-Checking.
  11. Metrics.
  12. Experimental Setup
  13. Models.
  14. Inference.
  15. Model Abstention.
  16. ...and 25 more sections

Figures (7)

  • Figure 1: Overview of WildHallucinations. We mine entities from real-world user queries and construct a knowledge source for these entities using web search. We prompt LLMs to generate information about the entities and automatically fact-check the outputs using the knowledge source. The texts colored in red are hallucinated by an LLM.
  • Figure 2: Left: Distribution over domains of entities. Right: Distribution over perplexities of entities.
  • Figure 3: WildFActScore-Strict of 15 models. Models are sorted based on their WildFActScore-Strict.
  • Figure 4: Comparison of WildFActScore-Strict with and without retrieval in RAG models.
  • Figure 5: WildFActScore-Strict for eight LLMs on entities grouped by perplexity values.
  • ...and 2 more figures