Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cultural Fidelity in Large-Language Models: An Evaluation of Online Language Resources as a Driver of Model Performance in Value Representation

Sharif Kazemi, Gloria Gerhardt, Jonty Katz, Caroline Ida Kuria, Estelle Pan, Umang Prabhakar

TL;DR

This study developed one of the largest and most robust datasets in this topic area with 21 country-language pairs, each of which contain 94 survey questions verified by native speakers, highlighting the link between LLM performance and digital data availability in target languages.

Abstract

The training data for LLMs embeds societal values, increasing their familiarity with the language's culture. Our analysis found that 44% of the variance in the ability of GPT-4o to reflect the societal values of a country, as measured by the World Values Survey, correlates with the availability of digital resources in that language. Notably, the error rate was more than five times higher for the languages of the lowest resource compared to the languages of the highest resource. For GPT-4-turbo, this correlation rose to 72%, suggesting efforts to improve the familiarity with the non-English language beyond the web-scraped data. Our study developed one of the largest and most robust datasets in this topic area with 21 country-language pairs, each of which contain 94 survey questions verified by native speakers. Our results highlight the link between LLM performance and digital data availability in target languages. Weaker performance in low-resource languages, especially prominent in the Global South, may worsen digital divides. We discuss strategies proposed to address this, including developing multilingual LLMs from the ground up and enhancing fine-tuning on diverse linguistic datasets, as seen in African language initiatives.

Cultural Fidelity in Large-Language Models: An Evaluation of Online Language Resources as a Driver of Model Performance in Value Representation

TL;DR

This study developed one of the largest and most robust datasets in this topic area with 21 country-language pairs, each of which contain 94 survey questions verified by native speakers, highlighting the link between LLM performance and digital data availability in target languages.

Abstract

The training data for LLMs embeds societal values, increasing their familiarity with the language's culture. Our analysis found that 44% of the variance in the ability of GPT-4o to reflect the societal values of a country, as measured by the World Values Survey, correlates with the availability of digital resources in that language. Notably, the error rate was more than five times higher for the languages of the lowest resource compared to the languages of the highest resource. For GPT-4-turbo, this correlation rose to 72%, suggesting efforts to improve the familiarity with the non-English language beyond the web-scraped data. Our study developed one of the largest and most robust datasets in this topic area with 21 country-language pairs, each of which contain 94 survey questions verified by native speakers. Our results highlight the link between LLM performance and digital data availability in target languages. Weaker performance in low-resource languages, especially prominent in the Global South, may worsen digital divides. We discuss strategies proposed to address this, including developing multilingual LLMs from the ground up and enhancing fine-tuning on diverse linguistic datasets, as seen in African language initiatives.

Paper Structure

This paper contains 41 sections, 1 equation, 13 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Literature review
  3. Performance of GPT in non-English languages
  4. GPT-4o
  5. GPT-4-turbo
  6. LLMs and societal value representation
  7. Methodology
  8. World Values Survey: selecting questions and languages
  9. Question selection
  10. Language selection
  11. Limitations and discrepancies
  12. Generation of Language Model Value Representation
  13. Accuracy of Value Representation: Comparison between Original and LLM Generated Values
  14. Low-Resource Languages
  15. Findings
  16. ...and 26 more sections

Figures (13)

  • Figure 1: Model architecture diagram: steps 1 to 3 of methodology
  • Figure 2: Societal representation error-rate vs. log number of websites, all country-language Pairs (GPT-4o)
  • Figure 3: Societal representation error-rate bands by country, mapped. 0-15% = low error-rate, 15-30% = medium error-rate, 30-45% = high error-rate (GPT-4o)
  • Figure 4: Societal representation error-rate vs. log number of websites, languages largely unique to that country (GPT-4o)
  • Figure 5: Error-rate of languages across different proportion thresholds for the error (GPT-4o)
  • ...and 8 more figures