Do Large Language Models Truly Understand Cross-cultural Differences?

TL;DR

The paper tackles the challenge of evaluating cross-cultural understanding in large language models by introducing SAGE, a scenario-based benchmark grounded in Cross-cultural Core Concepts (CCCs). It combines a three-layer, nine-category CCC taxonomy with four contextual interaction types to generate 4,530 culturally grounded items across Chinese and Spanish, plus potential cross-lingual extensions. Key findings show substantial zero-shot weaknesses in cross-cultural reasoning, a dimension hierarchy reflecting epistemic distance, and marked gains from concept knowledge injection and cross-lingual transfer, especially when using stronger prompt conditioning. The work provides a scalable, reproducible framework for assessing and guiding improvements in cross-cultural reasoning and language-agnostic alignment, with implications for instruction tuning and broader language coverage.

Abstract

In recent years, large language models (LLMs) have demonstrated strong performance on multilingual tasks. Given its wide range of applications, cross-cultural understanding capability is a crucial competency. However, existing benchmarks for evaluating whether LLMs genuinely possess this capability suffer from three key limitations: a lack of contextual scenarios, insufficient cross-cultural concept mapping, and limited deep cultural reasoning capabilities. To address these gaps, we propose SAGE, a scenario-based benchmark built via cross-cultural core concept alignment and generative task design, to evaluate LLMs' cross-cultural understanding and reasoning. Grounded in cultural theory, we categorize cross-cultural capabilities into nine dimensions. Using this framework, we curated 210 core concepts and constructed 4530 test items across 15 specific real-world scenarios, organized under four broader categories of cross-cultural situations, following established item design principles. The SAGE dataset supports continuous expansion, and experiments confirm its transferability to other languages. It reveals model weaknesses across both dimensions and scenarios, exposing systematic limitations in cross-cultural reasoning. While progress has been made, LLMs are still some distance away from reaching a truly nuanced cross-cultural understanding. In compliance with the anonymity policy, we include data and code in the supplement materials. In future versions, we will make them publicly available online.

Figures (8)

• Figure 1: Cross-cultural understanding is context-sensitive, necessitates cultural adaptation, and involves grasping the underlying values and cognitive frameworks behind cultural behaviors.
• Figure 2: SAGE Benchmark Design Framework.
• Figure 3: Three-layer taxonomy of Cross-cultural Core Concepts (CCC): Symbolic Layer, Behavioral & Ritual Layer, and Value Layer, each containing its respective conceptual categories. The taxonomy serves as the theoretical structure for CCC collection and analysis.
• Figure 4: Model-wise Reasoning Patterns
• Figure 5: Dimension-wise performance of GPT-4o under different levels of conceptual knowledge injection in Chinese (CN) and Spanish (ES). Results are shown across nine cultural dimensions, revealing injection responsiveness and cross-lingual generalization patterns.