Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-FAct: Assessing Factuality of Multilingual LLMs using FActScore

Sheikh Shafayat, Eunsu Kim, Juhyun Oh, Alice Oh

TL;DR

A simple pipeline for multilingual factuality evaluation is introduced, by applying FActScore (Min et al., 2023) for diverse languages, and comprehensive guidelines on multilingual factual evaluation for regionally diverse topics are provided.

Abstract

Evaluating the factuality of long-form large language model (LLM)-generated text is an important challenge. Recently there has been a surge of interest in factuality evaluation for English, but little is known about the factuality evaluation of multilingual LLMs, specially when it comes to long-form generation. %This paper systematically evaluates multilingual LLMs' factual accuracy across languages and geographic regions. We introduce a simple pipeline for multilingual factuality evaluation, by applying FActScore (Min et al., 2023) for diverse languages. In addition to evaluating multilingual factual generation, we evaluate the factual accuracy of long-form text generation in topics that reflect regional diversity. We also examine the feasibility of running the FActScore pipeline using non-English Wikipedia and provide comprehensive guidelines on multilingual factual evaluation for regionally diverse topics.

Multi-FAct: Assessing Factuality of Multilingual LLMs using FActScore

TL;DR

A simple pipeline for multilingual factuality evaluation is introduced, by applying FActScore (Min et al., 2023) for diverse languages, and comprehensive guidelines on multilingual factual evaluation for regionally diverse topics are provided.

Abstract

Evaluating the factuality of long-form large language model (LLM)-generated text is an important challenge. Recently there has been a surge of interest in factuality evaluation for English, but little is known about the factuality evaluation of multilingual LLMs, specially when it comes to long-form generation. %This paper systematically evaluates multilingual LLMs' factual accuracy across languages and geographic regions. We introduce a simple pipeline for multilingual factuality evaluation, by applying FActScore (Min et al., 2023) for diverse languages. In addition to evaluating multilingual factual generation, we evaluate the factual accuracy of long-form text generation in topics that reflect regional diversity. We also examine the feasibility of running the FActScore pipeline using non-English Wikipedia and provide comprehensive guidelines on multilingual factual evaluation for regionally diverse topics.
Paper Structure (32 sections, 1 equation, 8 figures, 8 tables)

This paper contains 32 sections, 1 equation, 8 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Factuality Evaluation
  4. Multilingual Factuality Evaluation
  5. Geo-culture biases of LLMs
  6. Multi-FAct Pipeline
  7. Generating Facts
  8. Model
  9. Language
  10. Prompt
  11. Translating Generated Facts
  12. Measuring Factuality
  13. Reliability of Multi-FAct
  14. Replication of original FActScore
  15. Effect of GPT-3.5 Translation in Multi-FAct
  16. ...and 17 more sections

Figures (8)

  • Figure 1: Our Multi-FAct Pipieline. The pipeline is structured into three main stages: 1) Obtaining multilingual generations, 2) Translating these facts into English using GPT-3.5, and 3) Measuring factuality, by breaking them down into smaller atomic facts and then verifying them by asking LLMs with context provided.
  • Figure 2: The Wikipedia size distribution for languages in this study. (c) shows the number of human-annotated examples that also have representation in the corresponding Wikipedia. Note that Wikipedia size differs widely across languages, which usually means non-English Wikipedia articles are not as comprehensive as English Wikipedia articles for fact verification.
  • Figure 3: Effect of using non-English Wikipedia as knowledge corpus for FActScore estimation. The x-axis in subfigure \ref{['subfig:length_vs_error']} represents the word count (in English) after translation. As the length of the non-English Wikipedia article increases, its utility for evaluating factuality increases.
  • Figure 4: Effect of corpus size on FActScore Estimation. Figure \ref{['subfig:wiki-len-extrapolate']} shows that comprehensive non-English sources might be used for factuality evaluation after translating into English.
  • Figure 5: FActScore of GPT-3.5 and GPT-4 for each language. Note that GPT-4 makes a strong jump from GPT-3.5, especially in low-resource languages.
  • ...and 3 more figures