Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LoFTI: Localization and Factuality Transfer to Indian Locales

Sona Elza Simon, Soumen Kumar Mondal, Abhishek Singhania, Sayambhu Sen, Preethi Jyothi

TL;DR

LoFTI addresses the gap in evaluating how LLMs transfer factual knowledge across locales by localizing global facts to Indian regions. It introduces a dataset construction pipeline involving entity-pair generation, reference text extraction, and localization with Mixtral; includes common-question generation and human validation. The study compares Mixtral, GPT-4, and two Mixtral-based variants (RARR and Revised), finding GPT-4 generally superior but its performance degrades with hyperlocality, while evidence-based Mixtral variants improve factuality. LoFTI is released under Apache 2.0 and serves as a benchmark for localized question answering and cross-locale factual transfer.

Abstract

Large language models (LLMs) encode vast amounts of world knowledge acquired via training on large web-scale datasets crawled from the internet. However, these datasets typically exhibit a geographical bias towards English-speaking Western countries. This results in LLMs producing biased or hallucinated responses to queries that require answers localized to other geographical regions. In this work, we introduce a new benchmark named LoFTI (Localization and Factuality Transfer to Indian Locales) that can be used to evaluate an LLM's localization and factual text transfer capabilities. LoFTI consists of factual statements about entities in source and target locations; the source locations are spread across the globe and the target locations are all within India with varying degrees of hyperlocality (country, states, cities). The entities span a wide variety of categories. We use LoFTI to evaluate Mixtral, GPT-4 and two other Mixtral-based approaches well-suited to the task of localized factual transfer. We demonstrate that LoFTI is a high-quality evaluation benchmark and all the models, including GPT-4, produce skewed results across varying levels of hyperlocality.

LoFTI: Localization and Factuality Transfer to Indian Locales

TL;DR

LoFTI addresses the gap in evaluating how LLMs transfer factual knowledge across locales by localizing global facts to Indian regions. It introduces a dataset construction pipeline involving entity-pair generation, reference text extraction, and localization with Mixtral; includes common-question generation and human validation. The study compares Mixtral, GPT-4, and two Mixtral-based variants (RARR and Revised), finding GPT-4 generally superior but its performance degrades with hyperlocality, while evidence-based Mixtral variants improve factuality. LoFTI is released under Apache 2.0 and serves as a benchmark for localized question answering and cross-locale factual transfer.

Abstract

Large language models (LLMs) encode vast amounts of world knowledge acquired via training on large web-scale datasets crawled from the internet. However, these datasets typically exhibit a geographical bias towards English-speaking Western countries. This results in LLMs producing biased or hallucinated responses to queries that require answers localized to other geographical regions. In this work, we introduce a new benchmark named LoFTI (Localization and Factuality Transfer to Indian Locales) that can be used to evaluate an LLM's localization and factual text transfer capabilities. LoFTI consists of factual statements about entities in source and target locations; the source locations are spread across the globe and the target locations are all within India with varying degrees of hyperlocality (country, states, cities). The entities span a wide variety of categories. We use LoFTI to evaluate Mixtral, GPT-4 and two other Mixtral-based approaches well-suited to the task of localized factual transfer. We demonstrate that LoFTI is a high-quality evaluation benchmark and all the models, including GPT-4, produce skewed results across varying levels of hyperlocality.
Paper Structure (47 sections, 13 figures, 15 tables)

This paper contains 47 sections, 13 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Methodology for Dataset Creation
  3. Generation of Entity-pairs
  4. Reference Text Generation
  5. Text Localization
  6. Common Question Generation
  7. Human Annotators
  8. Properties of LoFTI Dataset
  9. Category Distribution.
  10. Evaluation Metrics
  11. Entity Correctness
  12. Common Question Correctness
  13. Factual Correctness
  14. Models and Approaches
  15. Models
  16. ...and 32 more sections

Figures (13)

  • Figure 1: Illustration of LLM’s localized factual text transfer capabilities.
  • Figure 2: Illustration of the dataset creation pipeline with an example.
  • Figure 3: Illustrates the global distribution of the reference entities and the spread of target entities in India.
  • Figure A1: Few-shot prompt for common question generation on Mixtral
  • Figure A2: Prompt for text localization on Mixtral
  • ...and 8 more figures