Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models

Giwon Hong, Aryo Pradipta Gema, Rohit Saxena, Xiaotang Du, Ping Nie, Yu Zhao, Laura Perez-Beltrachini, Max Ryabinin, Xuanli He, Clémentine Fourrier, Pasquale Minervini

TL;DR

The paper tackles the problem of hallucinations in large language models by introducing the Hallucinations Leaderboard, an open framework that quantitatively benchmarks hallucination tendencies across factuality and faithfulness. It leverages the EleutherAI eval harness to assemble a diverse suite of 15 tasks spanning closed-book QA, fact-checking, summarisation, reading comprehension, and instruction following, evaluated on 20 open-source models in zero- to few-shot settings. A two-score system, factuality_score and faithfulness_score, aggregates performance across task groups, enabling cross-model comparisons and analysis of factors such as model family, size, and instruction tuning. Key findings show that instruction fine-tuning often enhances faithfulness but yields mixed gains in factuality, while larger models tend to improve both metrics, with some exception patterns; the framework promises a practical, open benchmark to guide model selection and development toward more reliable, information-aligned LLMs.

Abstract

Large Language Models (LLMs) have transformed the Natural Language Processing (NLP) landscape with their remarkable ability to understand and generate human-like text. However, these models are prone to ``hallucinations'' -- outputs that do not align with factual reality or the input context. This paper introduces the Hallucinations Leaderboard, an open initiative to quantitatively measure and compare the tendency of each model to produce hallucinations. The leaderboard uses a comprehensive set of benchmarks focusing on different aspects of hallucinations, such as factuality and faithfulness, across various tasks, including question-answering, summarisation, and reading comprehension. Our analysis provides insights into the performance of different models, guiding researchers and practitioners in choosing the most reliable models for their applications.

The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models

TL;DR

The paper tackles the problem of hallucinations in large language models by introducing the Hallucinations Leaderboard, an open framework that quantitatively benchmarks hallucination tendencies across factuality and faithfulness. It leverages the EleutherAI eval harness to assemble a diverse suite of 15 tasks spanning closed-book QA, fact-checking, summarisation, reading comprehension, and instruction following, evaluated on 20 open-source models in zero- to few-shot settings. A two-score system, factuality_score and faithfulness_score, aggregates performance across task groups, enabling cross-model comparisons and analysis of factors such as model family, size, and instruction tuning. Key findings show that instruction fine-tuning often enhances faithfulness but yields mixed gains in factuality, while larger models tend to improve both metrics, with some exception patterns; the framework promises a practical, open benchmark to guide model selection and development toward more reliable, information-aligned LLMs.

Abstract

Large Language Models (LLMs) have transformed the Natural Language Processing (NLP) landscape with their remarkable ability to understand and generate human-like text. However, these models are prone to ``hallucinations'' -- outputs that do not align with factual reality or the input context. This paper introduces the Hallucinations Leaderboard, an open initiative to quantitatively measure and compare the tendency of each model to produce hallucinations. The leaderboard uses a comprehensive set of benchmarks focusing on different aspects of hallucinations, such as factuality and faithfulness, across various tasks, including question-answering, summarisation, and reading comprehension. Our analysis provides insights into the performance of different models, guiding researchers and practitioners in choosing the most reliable models for their applications.
Paper Structure (24 sections, 3 figures, 7 tables)

This paper contains 24 sections, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Evaluation Framework
  3. Factuality Evaluation
  4. Closed-book Question Answering
  5. Fact-Checking
  6. Hallucination Detection
  7. Faithfulness Evaluation
  8. Summarisation
  9. Reading Comprehension
  10. Instruction Following
  11. Hallucination Detection
  12. Overall Evaluation Metrics
  13. Large Language Models
  14. Base Models
  15. Fine-tuned Models
  16. ...and 9 more sections

Figures (3)

  • Figure 1: Example of LLM factuality hallucination and factuality evaluation tasks in the Hallucination Leaderboard (left). Faithfulness hallucination example and tasks on the right.
  • Figure 2: The heatmap of results for various tasks of selected models. Each value in the heatmap follows the corresponding task's metric while clustering on the $x$-axis and $y$-axis was done after model/task normalisation.
  • Figure :