Language over Content: Tracing Cultural Understanding in Multilingual Large Language Models

TL;DR

This work investigates how multilingual LLMs internally represent culture by tracing activation-path circuits. By formalizing cultural queries as $Q_{L,C}$ and extracting internal paths $P(Q_{L,C})$, the authors compare path overlaps under two conditions to separate language-driven from culture-driven effects, using a BLEnD-based dataset extended across 49 language–culture configurations. The key finding is that internal path overlap is higher when questions share the same language, even across cultures, implying language-specific circuitry predominantly stores cultural knowledge; cross-language questions activate distinct internal paths, underscoring language as a strong organizing factor. An exception is the South/Korea–North Korea pair, which shows unusually low and variable overlaps in Korean, indicating complex interactions between language, politics, and culture. These results highlight the need for circuit-level analyses to understand cultural understanding in multilingual LLMs and suggest avenues for targeted interventions and broader multilingual evaluation.

Abstract

Large language models (LLMs) are increasingly used across diverse cultural contexts, making accurate cultural understanding essential. Prior evaluations have mostly focused on output-level performance, obscuring the factors that drive differences in responses, while studies using circuit analysis have covered few languages and rarely focused on culture. In this work, we trace LLMs' internal cultural understanding mechanisms by measuring activation path overlaps when answering semantically equivalent questions under two conditions: varying the target country while fixing the question language, and varying the question language while fixing the country. We also use same-language country pairs to disentangle language from cultural aspects. Results show that internal paths overlap more for same-language, cross-country questions than for cross-language, same-country questions, indicating strong language-specific patterns. Notably, the South Korea-North Korea pair exhibits low overlap and high variability, showing that linguistic similarity does not guarantee aligned internal representation.

Figures (4)

• Figure 1: Overview of Tracing Cultural Understanding in Multilingual Large Language Models.
• Figure 2: (a) Path overlap by country pair when the question language is fixed; We find that overlaps remain relatively high, with linguistically similar country pairs showing especially high reuse of internal paths. (b) Path overlap by language pair when the target culture is fixed; We find that overlaps drop markedly when the query language changes, indicating that language (rather than meaning) dominates internal path selection. Each bar shows the mean (±95% CI), sorted in descending order; hatched bars denote linguistically similar pairs, and the orange horizontal line marks the overall average.
• Figure 3: Path overlap between questions on South and North Korean culture by question language. We find that path overlaps are low in Korean languages than in non-Korean languages.
• Figure 4: Path overlap between questions for similar-language pairs under a fixed target culture. US-UK and Spain-Mexico show high and stable overlap while South Korea-North Korea shows lower and more variable overlap.