Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding Cultural Alignment in Multilingual LLMs via Natural Debate Statements

Vlad-Andrei Negru, Camelia Lemnaru, Mihai Surdeanu, Rodica Potolea

TL;DR

It is found that culturally-distinct LLMs reflect the values and norms of the countries in which they were developed, highlighting their inability to adapt to the sociocultural backgrounds of their users.

Abstract

In this work we investigate the sociocultural values learned by large language models (LLMs). We introduce a novel open-access dataset, Sociocultural Statements, constructed from natural debate statements using a multi-step methodology. The dataset is synthetically labeled to enable the quantization of sociocultural norms and beliefs that LLMs exhibit in their responses to these statements, according to the Hofstede cultural dimensions. We verify the accuracy of synthetic labels using human quality control on a representative sample. We conduct a comparative analysis between two groups of LLMs developed in different countries (U.S. and China), and use as a comparative baseline patterns observed in human measurements. Using this new dataset and the analysis above, we found that culturally-distinct LLMs reflect the values and norms of the countries in which they were developed, highlighting their inability to adapt to the sociocultural backgrounds of their users.

Understanding Cultural Alignment in Multilingual LLMs via Natural Debate Statements

TL;DR

It is found that culturally-distinct LLMs reflect the values and norms of the countries in which they were developed, highlighting their inability to adapt to the sociocultural backgrounds of their users.

Abstract

In this work we investigate the sociocultural values learned by large language models (LLMs). We introduce a novel open-access dataset, Sociocultural Statements, constructed from natural debate statements using a multi-step methodology. The dataset is synthetically labeled to enable the quantization of sociocultural norms and beliefs that LLMs exhibit in their responses to these statements, according to the Hofstede cultural dimensions. We verify the accuracy of synthetic labels using human quality control on a representative sample. We conduct a comparative analysis between two groups of LLMs developed in different countries (U.S. and China), and use as a comparative baseline patterns observed in human measurements. Using this new dataset and the analysis above, we found that culturally-distinct LLMs reflect the values and norms of the countries in which they were developed, highlighting their inability to adapt to the sociocultural backgrounds of their users.
Paper Structure (18 sections, 3 figures, 4 tables)

This paper contains 18 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. The Hofstede cultural dimensions
  4. Dataset construction and alignment
  5. Statement extraction and filtering
  6. Statement scoring
  7. Human quality control
  8. Evaluation Methodology
  9. Experimental results
  10. Experimental settings
  11. Results and discussion
  12. Comparing U.S. vs. Chinese LLMs
  13. Comparing individual models on their "native" languages
  14. Comparing individual models across different languages
  15. Potential explanations
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Scoring of humans and LLMs according to the Hofstede dimensions. LLMs were prompted to agree or disagree with a given statement, and each response was assigned a polarity score ranging from $-2$ to $+2$ along the corresponding Hofstede dimension. These scores were then averaged and linearly mapped to a 0--100 scale to ensure comparability with human measurements. We observe that the LLMs largely follow the trends observed in human measurements across all six dimensions.
  • Figure 2: The statement scoring pipeline based on the six Hofstede dimensions. We collect natural debate statements via web scraping and filter them based on vote counts. Step 1 further removes statements that are irrelevant to the Hofstede dimensions. Step 2 assigns a score ($-2$ to $+2$), indicating how agreement with a statement aligns with the negative or positive polarity of each Hofstede dimension. Each synthetic labeling step is manually verified on a representative sample.
  • Figure 3: Average scores across all Hofstede dimensions for each model when prompted in its "native" language, evaluated on Sociocultural Statements dataset. The red bars represent Chinese models, while the blue bars represent U.S. models. Despite values being in the interval [-2, 2], we used [-1, 1] for the axis as it fits the average scores.