Can LLMs Grasp Implicit Cultural Values? Benchmarking LLMs' Cultural Intelligence with CQ-Bench

TL;DR

CQ-Bench introduces a dedicated benchmark to evaluate LLMs’ ability to infer implicit cultural values from conversational contexts, addressing a gap not captured by prior culture-focused tasks. The authors build an automatic, multi-turn dialogue generation pipeline with rigorous validation (GPT-4o-based incorporation/consistency checks and human evaluation), yielding high agreement with human judgments. They define three escalating tasks—Attitude Detection, Value Selection, and Value Extraction—and assess a wide range of models, showing that larger models generally perform better but still struggle with nuanced attitudes and religious values. Importantly, targeted fine-tuning on a small set of culturally rich examples significantly boosts performance for smaller models, indicating effective distillation of cultural reasoning. The work highlights practical pathways to improve cross-cultural reasoning in LLMs and provides a scalable framework for future cultural intelligence research.

Abstract

Cultural Intelligence (CQ) refers to the ability to understand unfamiliar cultural contexts, a crucial skill for large language models (LLMs) to effectively engage with globally diverse users. Existing studies often focus on explicitly stated cultural norms, but fail to capture the subtle, implicit values that are common in daily conversation. To address this gap, we introduce CQBench, a benchmark specifically designed to assess LLMs' capability to infer implicit cultural values from natural conversational contexts. CQBench consists of multi character conversation based stories using values from the World Value Survey and the GlobalOpinions, with topics including ethical, religious, social, etc. Our automatic dataset construction pipeline integrates rigorous validation procedures (incorporation, consistency, and implicitness checks), achieving a 94.5% human model agreement in the final validation. To leverage CQBench data, we design three tasks of increasing complexity: attitude detection, value selection, and value extraction. These tasks evaluate whether models can detect attitude and recognize values embedded within natural dialogues rather than relying on explicit cultural knowledge. We find that while frontier models like o1 reach human level performance in value selection (0.809 F1), they still fall short in nuanced attitude detection (0.622 F1). Notably, finetuning a smaller LLaMA-3.2-3B on only 500 culturally rich examples improves performance by over 10%, even outperforming o3-mini in some cases. Using CQ-Bench, we provide insights into the current challenges in LLMs' CQ research and suggest practical pathways for enhancing LLMs' cross-cultural reasoning abilities.

Figures (8)

• Figure 1: An illustration of CQ-Bench. We construct three distinct tasks based on conversation-style stories to assess the cultural intelligence of LLMs in CQ-Bench.
• Figure 2: Dataset construction pipeline. We first create value sets, and then generate multi-character conversation style story. We conduct multiple checks and refinement to improve the quality of the story.
• Figure 3: Category specific results. Overall, models perform worst in the Religious setting, and category-specific datasets yield higher scores than randomly sampled ones.
• Figure 4: Screenshot of human annotation guideline (1)
• Figure 5: Screenshot of human annotation guideline (2)