XCR-Bench: A Multi-Task Benchmark for Evaluating Cultural Reasoning in LLMs

TL;DR

XCR-Bench presents a comprehensive cross-cultural reasoning benchmark that ties Newmark's Culture-Specific Items to Hall's Iceberg Model, enabling fine-grained evaluation of LLMs on CSI identification, prediction, and adaptation across intra- and inter-lingual contexts. The authors construct a sizeable, human-annotated parallel corpus (4,136 sentences, 1,098 CSIs across 7 categories) spanning Western CSIs and culturally adapted equivalents for Chinese, Arabic, and Bengali (WB and BD), with rigorous annotation and adjudication. Empirical results show persistent weaknesses in cross-cultural CSI handling and reveal regional biases within a single language during adaptation, especially for Bengali, underscoring the need for broader cultural coverage and improved prompting strategies. The dataset and accompanying metrics contribute a practical resource for advancing cross-cultural NLP and bias analysis, with clear avenues for future expansion and methodological refinement.

Abstract

Cross-cultural competence in large language models (LLMs) requires the ability to identify Culture-Specific Items (CSIs) and to adapt them appropriately across cultural contexts. Progress in evaluating this capability has been constrained by the scarcity of high-quality CSI-annotated corpora with parallel cross-cultural sentence pairs. To address this limitation, we introduce XCR-Bench, a Cross(X)-Cultural Reasoning Benchmark consisting of 4.9k parallel sentences and 1,098 unique CSIs, spanning three distinct reasoning tasks with corresponding evaluation metrics. Our corpus integrates Newmark's CSI framework with Hall's Triad of Culture, enabling systematic analysis of cultural reasoning beyond surface-level artifacts and into semi-visible and invisible cultural elements such as social norms, beliefs, and values. Our findings show that state-of-the-art LLMs exhibit consistent weaknesses in identifying and adapting CSIs related to social etiquette and cultural reference. Additionally, we find evidence that LLMs encode regional and ethno-religious biases even within a single linguistic setting during cultural adaptation. We release our corpus and code to facilitate future research on cross-cultural NLP.

Figures (13)

• Figure 1: XCR-Bench corpus creation and annotation pipeline. (1) Culture-specific items (CSIs) and their contexts are extracted from well-known cultural databases. (2) Sentences are generated using LLMs and filtered for accuracy and fluency. (3) Each sentence is annotated with a CSI category and mapped to Hall's cultural levels.
• Figure 2: Performance of LLMs across CSI categories for CSI Identification and Prediction tasks. For clarity, only soft evaluation metrics are visualized. The results reveal consistent difficulties in CSI Identification across CSI categories.
• Figure 3: Mapping between CSI categories and Hall's cultural levels. The textured bar plots indicate a higher concentration of semi-visible and invisible cultural elements across all CSI categories.
• Figure 4: Mapping between CSI categories and Hall's cultural elements in the XCR-Bench Corpus.
• Figure 5: Performance of LLMs across Hall's Triad of Culture (visible, semi-visible, and invisible) for CSI Identification. The HI-CSI metric exhibits the lowest overall performance for Semi-visible level, indicating that LLMs struggle with exact CSI identification for this level. In contrast, SI-CSI scores reveal a systematic performance decline from Visible to Invisible cultural levels.