Neuron-Level Analysis of Cultural Understanding in Large Language Models

TL;DR

This work introduces CULNIG, a gradient-based neuron attribution pipeline to dissect cultural understanding in large language models. It identifies culture-general neurons (less than 1% of neurons, concentrated in shallow-to-middle MLP layers) and culture-specific neurons tied to individual cultures, demonstrating that masking these neurons degrades cultural benchmarks significantly with minimal impact on general NLU tasks. Culture-general neurons exhibit broad, cross-cultural influence, while culture-specific neurons affect target cultures and related ones, revealing shared representations across cultures. The study also shows that fine-tuning with NLU data can erode cultural understanding if modules rich in culture-general neurons are updated, offering practical guidance for efficient, robust model training and engineering.

Abstract

As large language models (LLMs) are increasingly deployed worldwide, ensuring their fair and comprehensive cultural understanding is important. However, LLMs exhibit cultural bias and limited awareness of underrepresented cultures, while the mechanisms underlying their cultural understanding remain underexplored. To fill this gap, we conduct a neuron-level analysis to identify neurons that drive cultural behavior, introducing a gradient-based scoring method with additional filtering for precise refinement. We identify both culture-general neurons contributing to cultural understanding regardless of cultures, and culture-specific neurons tied to an individual culture. These neurons account for less than 1% of all neurons and are concentrated in shallow to middle MLP layers. We validate their role by showing that suppressing them substantially degrades performance on cultural benchmarks (by up to 30%), while performance on general natural language understanding (NLU) benchmarks remains largely unaffected. Moreover, we show that culture-specific neurons support knowledge of not only the target culture, but also related cultures. Finally, we demonstrate that training on NLU benchmarks can diminish models' cultural understanding when we update modules containing many culture-general neurons. These findings provide insights into the internal mechanisms of LLMs and offer practical guidance for model training and engineering. Our code is available at https://github.com/ynklab/CULNIG

Figures (58)

• Figure 1: An overview of CULNIG when identifying culture-general neurons. We first select the top $t\%$ of the neurons ranked by gradient-based attribution scores on $\text{BLEnD}_{\text{neur}} - \text{BLEnD}_{\text{ctrl}}$ ($s_{\text{neur}} - s_{\text{ctrl}}$) to find neurons contributing to cultural mechanisms. By subtracting $s_{\text{ctrl}}$, we exclude neurons facilitating task understanding. We then remove the top $r\%$ of the neurons on $\text{CRC}_\text{neur}$ to filter out superficial neurons activated by country names.
• Figure 2: MLP modules
• Figure 3: attention modules
• Figure 5: The distribution of culture-general neurons in gemma-3-12b-it.
• Figure 6: $\text{BLEnD}_{\text{test}}$