Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking

Yi Fung, Ruining Zhao, Jae Doo, Chenkai Sun, Heng Ji

TL;DR

The paper tackles cultural bias and the lack of fine-grained cross-cultural commonsense knowledge in large language models by introducing CultureAtlas, a massively multicultural benchmark built from Wikipedia and linked pages that yields self-contained cultural assertions and fine-grained culture profiles. It combines bottom-up web-derived data with top-down topic guidance, generating high-quality positive and adversarial negative samples, with rigorous quality checks yielding over 90% pass rates. Benchmark experiments across open and closed LLMs reveal that Vicuna-type backbones often outperform Llama-2 at similar scales, alignment via human feedback yields mixed gains, and model size correlates with cultural reasoning ability; performance varies significantly by topic and cultural-resource level. The work provides a scalable resource and evaluation framework to foster culturally inclusive and debiased LLMs, and outlines future directions including multimedia collaboration and broader low-resource language coverage.

Abstract

Pretrained large language models have revolutionized many applications but still face challenges related to cultural bias and a lack of cultural commonsense knowledge crucial for guiding cross-culture communication and interactions. Recognizing the shortcomings of existing methods in capturing the diverse and rich cultures across the world, this paper introduces a novel approach for massively multicultural knowledge acquisition. Specifically, our method strategically navigates from densely informative Wikipedia documents on cultural topics to an extensive network of linked pages. Leveraging this valuable source of data collection, we construct the CultureAtlas dataset, which covers a wide range of sub-country level geographical regions and ethnolinguistic groups, with data cleaning and preprocessing to ensure textual assertion sentence self-containment, as well as fine-grained cultural profile information extraction. Our dataset not only facilitates the evaluation of language model performance in culturally diverse contexts but also serves as a foundational tool for the development of culturally sensitive and aware language models. Our work marks an important step towards deeper understanding and bridging the gaps of cultural disparities in AI, to promote a more inclusive and balanced representation of global cultures in the digital domain.

Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking

TL;DR

The paper tackles cultural bias and the lack of fine-grained cross-cultural commonsense knowledge in large language models by introducing CultureAtlas, a massively multicultural benchmark built from Wikipedia and linked pages that yields self-contained cultural assertions and fine-grained culture profiles. It combines bottom-up web-derived data with top-down topic guidance, generating high-quality positive and adversarial negative samples, with rigorous quality checks yielding over 90% pass rates. Benchmark experiments across open and closed LLMs reveal that Vicuna-type backbones often outperform Llama-2 at similar scales, alignment via human feedback yields mixed gains, and model size correlates with cultural reasoning ability; performance varies significantly by topic and cultural-resource level. The work provides a scalable resource and evaluation framework to foster culturally inclusive and debiased LLMs, and outlines future directions including multimedia collaboration and broader low-resource language coverage.

Abstract

Pretrained large language models have revolutionized many applications but still face challenges related to cultural bias and a lack of cultural commonsense knowledge crucial for guiding cross-culture communication and interactions. Recognizing the shortcomings of existing methods in capturing the diverse and rich cultures across the world, this paper introduces a novel approach for massively multicultural knowledge acquisition. Specifically, our method strategically navigates from densely informative Wikipedia documents on cultural topics to an extensive network of linked pages. Leveraging this valuable source of data collection, we construct the CultureAtlas dataset, which covers a wide range of sub-country level geographical regions and ethnolinguistic groups, with data cleaning and preprocessing to ensure textual assertion sentence self-containment, as well as fine-grained cultural profile information extraction. Our dataset not only facilitates the evaluation of language model performance in culturally diverse contexts but also serves as a foundational tool for the development of culturally sensitive and aware language models. Our work marks an important step towards deeper understanding and bridging the gaps of cultural disparities in AI, to promote a more inclusive and balanced representation of global cultures in the digital domain.
Paper Structure (21 sections, 3 figures, 10 tables)

This paper contains 21 sections, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Benchmark Construction
  3. Data Collection and Data Preprocessing
  4. Positive Data Samples
  5. Negative Data Samples
  6. Quality Check on Data
  7. Descriptive Stats
  8. Training-Testing Data Split
  9. Experiments
  10. Task Setting
  11. Model Setup
  12. Results
  13. Ablation on the Challenging Setting brought by Fine-Grained Cultural Knowledge Frame Profiling
  14. Error Analysis
  15. Related Work
  16. ...and 6 more sections

Figures (3)

  • Figure 1: An example illustration of different cultural practices, with regards to Chinese New Year red envelopes across different geographical regions such as Beijing/Shanghai versus HK/Macau/Guangdong.
  • Figure 2: An overarching view of our CultureAtlas benchmark construction process.
  • Figure 3: LLM performance by Topic (result from a single LM backbone ad-hoc, llama2-7b).