Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CulturePark: Boosting Cross-cultural Understanding in Large Language Models

Cheng Li, Damien Teney, Linyi Yang, Qingsong Wen, Xing Xie, Jindong Wang

TL;DR

CulturePark is introduced, an LLM-powered multi-agent communication framework for cultural data collection and proves an important step in addressing cultural bias and advancing the democratization of AI, highlighting the critical role of culturally inclusive data in model training.

Abstract

Cultural bias is pervasive in many large language models (LLMs), largely due to the deficiency of data representative of different cultures. Typically, cultural datasets and benchmarks are constructed either by extracting subsets of existing datasets or by aggregating from platforms such as Wikipedia and social media. However, these approaches are highly dependent on real-world data and human annotations, making them costly and difficult to scale. Inspired by cognitive theories on social communication, this paper introduces CulturePark, an LLM-powered multi-agent communication framework for cultural data collection. CulturePark simulates cross-cultural human communication with LLM-based agents playing roles in different cultures. It generates high-quality cross-cultural dialogues encapsulating human beliefs, norms, and customs. Using CulturePark, we generated 41,000 cultural samples to fine-tune eight culture-specific LLMs. We evaluated these models across three downstream tasks: content moderation, cultural alignment, and cultural education. Results show that for content moderation, our GPT-3.5-based models either match or outperform GPT-4 on datasets. Regarding cultural alignment, our models surpass GPT-4 on Hofstede's VSM 13 framework. Furthermore, for cultural education of human participants, our models demonstrate superior outcomes in both learning efficacy and user experience compared to GPT-4. CulturePark proves an important step in addressing cultural bias and advancing the democratization of AI, highlighting the critical role of culturally inclusive data in model training. Code is released at https://github.com/Scarelette/CulturePark.

CulturePark: Boosting Cross-cultural Understanding in Large Language Models

TL;DR

CulturePark is introduced, an LLM-powered multi-agent communication framework for cultural data collection and proves an important step in addressing cultural bias and advancing the democratization of AI, highlighting the critical role of culturally inclusive data in model training.

Abstract

Cultural bias is pervasive in many large language models (LLMs), largely due to the deficiency of data representative of different cultures. Typically, cultural datasets and benchmarks are constructed either by extracting subsets of existing datasets or by aggregating from platforms such as Wikipedia and social media. However, these approaches are highly dependent on real-world data and human annotations, making them costly and difficult to scale. Inspired by cognitive theories on social communication, this paper introduces CulturePark, an LLM-powered multi-agent communication framework for cultural data collection. CulturePark simulates cross-cultural human communication with LLM-based agents playing roles in different cultures. It generates high-quality cross-cultural dialogues encapsulating human beliefs, norms, and customs. Using CulturePark, we generated 41,000 cultural samples to fine-tune eight culture-specific LLMs. We evaluated these models across three downstream tasks: content moderation, cultural alignment, and cultural education. Results show that for content moderation, our GPT-3.5-based models either match or outperform GPT-4 on datasets. Regarding cultural alignment, our models surpass GPT-4 on Hofstede's VSM 13 framework. Furthermore, for cultural education of human participants, our models demonstrate superior outcomes in both learning efficacy and user experience compared to GPT-4. CulturePark proves an important step in addressing cultural bias and advancing the democratization of AI, highlighting the critical role of culturally inclusive data in model training. Code is released at https://github.com/Scarelette/CulturePark.
Paper Structure (47 sections, 7 equations, 13 figures, 10 tables)

This paper contains 47 sections, 7 equations, 13 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Cultural bias in LLMs
  4. Cultural benchmarks and datasets
  5. Existing solutions to cultural bias
  6. CulturePark
  7. Design
  8. Data refinement and fine-tuning
  9. CulturePark benefits cultural understanding and fine-tuning
  10. Experiments
  11. Evaluation on content moderation tasks
  12. Setup.
  13. Main Results.
  14. Ablation study.
  15. Evaluation on cultural alignment via Hofstede's Cultural Dimensions Theory
  16. ...and 32 more sections

Figures (13)

  • Figure 1: CulturePark is an LLM-based multi-agent communication platform for cultural data collection. Leveraging CulturePark, we can collect a cross-cultural dialogue dataset, which can then be used for fine-tuning culturally specific LLMs to be applied to different downstream tasks: content moderation, cultural alignment, and cultural education.
  • Figure 2: Cross-cultural dialogue and data refinement for fine-tuning LLMs using CulturePark.
  • Figure 3: Results on content moderation.
  • Figure 4: Results on culture alignment via Hofstede's Cultural Dimensions Theory.
  • Figure 5: More discussions on CulturePark.
  • ...and 8 more figures