From Word to World: Evaluate and Mitigate Culture Bias in LLMs via Word Association Test

TL;DR

This work evaluates cross-cultural cognition in LLMs using a human-inspired Word Association Test (WAT) adapted for LLMs and introduces CultureSteer, a culture-aware steering mechanism that subtly reshapes internal semantic representations to align with culture-specific spaces. The authors show persistent Western biases in baseline LLMs at the word-association level and demonstrate that CultureSteer improves cross-cultural alignment, outperforming prompt-based approaches and culture-specific models in WAT and downstream tasks. A new Position-Weighted Recall metric (PWR@K) is proposed to capture the importance of early-ranked associations, complemented by a DCG@K-style analysis to validate ranking robustness. The results suggest that integrating culture-specific semantic directions into the embedding space yields more inclusive, culturally aware language technologies with broad generalization to culture-related applications.

Abstract

The human-centered word association test (WAT) serves as a cognitive proxy, revealing sociocultural variations through culturally shared semantic expectations and implicit linguistic patterns shaped by lived experiences. We extend this test into an LLM-adaptive, free-relation task to assess the alignment of large language models (LLMs) with cross-cultural cognition. To address culture preference, we propose CultureSteer, an innovative approach that moves beyond superficial cultural prompting by embedding cultural-specific semantic associations directly within the model's internal representation space. Experiments show that current LLMs exhibit significant bias toward Western (notably American) schemas at the word association level. In contrast, our model substantially improves cross-cultural alignment, capturing diverse semantic associations. Further validation on culture-sensitive downstream tasks confirms its efficacy in fostering cognitive alignment across cultures. This work contributes a novel methodological paradigm for enhancing cultural awareness in LLMs, advancing the development of more inclusive language technologies.

Figures (5)

• Figure 1: Cross-cultural comparison of word associations for the cue "Red" between LLM predictions and human responses.
• Figure 2: The framework of CultureSteer model. Top: model pipeline; Bottom: training and inference process based on WAT.
• Figure 3: Fine-grained performance comparison across 22 semantic classes in the test set with PWR@20. Red denotes Global Knowledge, green denotes Perceptual Experience, and blue denotes Cultural Ideologies. Other PWR@K (K=3, 5, 10) results are shown in Appendix \ref{['app:radars']}.
• Figure 4: Probability differences between CultureSteer and vanilla models across USA, UK and OC. Red marks denote culture-specific words, while gray marks represent general words common across cultures.
• Figure 5: Graunalr performance in PWR@3, 5 and 10.