Tears or Cheers? Benchmarking LLMs via Culturally Elicited Distinct Affective Responses

TL;DR

CEDAR introduces a multilingual, multimodal benchmark designed to probe culturally elicited affective responses. It combines an LLM-driven preselection pipeline with rigorous native-language annotation to produce $10{,}962$ instances across $7$ languages and $14$ emotion categories, spanning both image-grounded and text-only data. Evaluations of $17$ representative multilingual models reveal a dissociation between language consistency and cultural alignment, with a pervasive high-arousal bias and substantial cross-language variation, underscoring the need for culture-aware affective understanding in LLMs. Cedar provides a scalable, ethically curated resource for studying cross-cultural affective reasoning and motivates future work on multilingual alignment and multimodal emotion interpretation.

Abstract

Culture serves as a fundamental determinant of human affective processing and profoundly shapes how individuals perceive and interpret emotional stimuli. Despite this intrinsic link extant evaluations regarding cultural alignment within Large Language Models primarily prioritize declarative knowledge such as geographical facts or established societal customs. These benchmarks remain insufficient to capture the subjective interpretative variance inherent to diverse sociocultural lenses. To address this limitation, we introduce CEDAR, a multimodal benchmark constructed entirely from scenarios capturing Culturally \underline{\textsc{E}}licited \underline{\textsc{D}}istinct \underline{\textsc{A}}ffective \underline{\textsc{R}}esponses. To construct CEDAR, we implement a novel pipeline that leverages LLM-generated provisional labels to isolate instances yielding cross-cultural emotional distinctions, and subsequently derives reliable ground-truth annotations through rigorous human evaluation. The resulting benchmark comprises 10,962 instances across seven languages and 14 fine-grained emotion categories, with each language including 400 multimodal and 1,166 text-only samples. Comprehensive evaluations of 17 representative multilingual models reveal a dissociation between language consistency and cultural alignment, demonstrating that culturally grounded affective understanding remains a significant challenge for current models.

Figures (10)

• Figure 1: A representative example of culturally distinct scenarios from Cedar. The figure illustrates how identical multimodal (top) and text-only (bottom) inputs are mapped to different language-specific ground truths.
• Figure 2: Statistics of Cedar.
• Figure 3: Visualization of GEPP across multimodal and text-only subsets. The scatter points represent the individual performance of the evaluated LLMs for each emotion category.
• Figure 4: LSEPP for four illustrative models on multimodal and text-only subsets. Each axis represents the LSEPP value for a specific emotion, demonstrating the variability of bias scores across seven languages.
• Figure 5: Model comparison based on the GRQB. The chart displays the GRQB of model predictions across the four quadrants for multimodal and text-only subsets.