Probing Pre-Trained Language Models for Cross-Cultural Differences in Values
Arnav Arora, Lucie-Aimée Kaffee, Isabelle Augenstein
TL;DR
The paper investigates whether multilingual pretrained language models embed cross-cultural values and whether these embeddings align with established value surveys. It introduces a probing framework that converts Hofstede and World Values Survey items into cloze-style prompts and evaluates three multilingual models across 13 languages using mask-based predictions and Spearman correlations. Findings show that PTLMs reveal cross-cultural value differences, but these signals rarely align with Hofstede or WVS benchmarks and vary considerably across models, highlighting sensitivity to training data and model design. The work emphasizes the need for diverse, culturally representative data and careful auditing when deploying models in cross-cultural contexts, and it outlines future directions for improving alignment between learned representations and sociocultural theories.
Abstract
Language embeds information about social, cultural, and political values people hold. Prior work has explored social and potentially harmful biases encoded in Pre-Trained Language models (PTLMs). However, there has been no systematic study investigating how values embedded in these models vary across cultures. In this paper, we introduce probes to study which values across cultures are embedded in these models, and whether they align with existing theories and cross-cultural value surveys. We find that PTLMs capture differences in values across cultures, but those only weakly align with established value surveys. We discuss implications of using mis-aligned models in cross-cultural settings, as well as ways of aligning PTLMs with value surveys.