Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Chronicles of RiDiC: Generating Datasets with Controlled Popularity Distribution for Long-form Factuality Evaluation

Pavel Braslavski, Dmitrii Iarosh, Nikita Sushko, Andrey Sakhovskiy, Vasily Konovalov, Elena Tutubalina, Alexander Panchenko

Abstract

We present a configurable pipeline for generating multilingual sets of entities with specified characteristics, such as domain, geographical location and popularity, using data from Wikipedia and Wikidata. These datasets are intended for evaluating the factuality of LLMs' long-form generation, thereby complementing evaluation based on short-form QA datasets. We present the RiDiC dataset as an example of this approach. RiDiC contains 3,000 entities from three domains -- rivers, natural disasters, and car models -- spanning different popularity tiers. Each entity is accompanied by its geographical location, English and Chinese names (if available) and relevant English and Chinese Wikipedia content, which is used to evaluate LLMs' responses. Generations about RiDiC entities were obtained from three LLMs in English and Chinese. These were then evaluated using a third-party factuality checker, which showed that entities from our dataset caused even frontier models to hallucinate. To facilitate the evaluation of LLMs' long-form factuality in multiple languages, the code, data, and generation/evaluation scripts have been released.

The Chronicles of RiDiC: Generating Datasets with Controlled Popularity Distribution for Long-form Factuality Evaluation

Abstract

We present a configurable pipeline for generating multilingual sets of entities with specified characteristics, such as domain, geographical location and popularity, using data from Wikipedia and Wikidata. These datasets are intended for evaluating the factuality of LLMs' long-form generation, thereby complementing evaluation based on short-form QA datasets. We present the RiDiC dataset as an example of this approach. RiDiC contains 3,000 entities from three domains -- rivers, natural disasters, and car models -- spanning different popularity tiers. Each entity is accompanied by its geographical location, English and Chinese names (if available) and relevant English and Chinese Wikipedia content, which is used to evaluate LLMs' responses. Generations about RiDiC entities were obtained from three LLMs in English and Chinese. These were then evaluated using a third-party factuality checker, which showed that entities from our dataset caused even frontier models to hallucinate. To facilitate the evaluation of LLMs' long-form factuality in multiple languages, the code, data, and generation/evaluation scripts have been released.

Paper Structure

This paper contains 21 sections, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Generation Pipeline
  4. RiDiC Dataset
  5. Experiments
  6. Factuality Evaluation
  7. Evaluation Methodology
  8. Implementation Details
  9. Factuality Evaluation in Chinese
  10. LLMs' Responses
  11. Factuality Evaluation Results
  12. LLMs Hallucinate on Rare Entities More
  13. Low Factuality in Chinese
  14. Factuality is Sensitive to Domain
  15. LLMs Exhibit no Clear Geographic Bias
  16. ...and 6 more sections

Figures (2)

  • Figure 1: The process of generating a dataset begins with a Wikidata SPARQL query that defines the class of interest. Then, for each class member, attributes and popularity statistics are gathered from Wikidata and Wikipedia. The dataset is formed by sampling the required number of entities with the desired popularity distribution; Wikipedia content is collected as evidence. Once the dataset is complete, the LLMs' generated content about the collected entities is evaluated using a factuality checker.
  • Figure 2: Vocabulary growth in Qwen's generations about disasters in different popularity tiers.